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1.0  SUMMARY 


Meeting  the  demand  for  top  technical  cybersecurity  talent  is  one  of  the  difficult 
challenges  facing  military  and  civilian  government  leaders.  Competitions  have  demonstrated 
promising  avenues  for  identification  of  future  talent,  for  ranking  candidates  on  cyber  skills,  and 
for  motivating  candidates  to  become  fully  committed  to  advancing  their  skills  in  cybersecurity. 
The  US  Cyber  Challenge  (USCC)  is  helping  to  develop  the  next  generation  of  cyber  experts 
through  education  and  hands-on  defense  gaming  strategies.  By  creating  excitement  and 
highlighting  competitors’  successes,  the  USCC  attempts  to: 

•  Demonstrate  there  are  cool  jobs  in  technology; 

•  Provide  recognition  for  achievements; 

•  Create  a  hierarchy  of  talent  challenges; 

•  Organize  highly  selective  summer  camps; 

•  Ensure  the  public  knows  how  impressive  it  is  to  be  selected  to  participate;  and 

•  Analyze  what  works  and  what  does  not  work  in  each  of  these  areas. 

During  this  past  period  of  performance,  September  2010  through  October  2011,  the 
USCC  held  several  competitions  and  challenges  and  also  developed  the  initial  assessment 
framework  to  measure  the  performance  outcomes  of  the  program.  The  impact  of  the  USCC  is 
being  measured  against  this  assessment  framework  and  is  being  measured  by  the  increasing 
number  of  participants  in  the  USCC  activities.  For  example,  for  summer  camps  held  in  2010, 
our  initial  pilot  effort  involving  three  universities  and  55  participants  to  this  summer  camps 
which  involved  6  universities  and  230  participants. 

In  summary,  there  are  two  key  data  points  to  highlight  from  this  past  year’s  data 
collection  and  assessment  effort: 

•  For  student  vetting,  “Simply  identifying  students  who  were  likely  to  engage  increased  the 
overall  value  of  the  experience.”  This  statement  illustrates  the  use  of  competitions  such 
as  the  on-line  competitions  of  Cyber  Quests  for  the  1 8  years  and  older  participants  and 
the  Cyber  Foundations  for  the  high  school  participants  are  reaching  the  targeted 
population  and  addressing  the  short-term  objectives  of  identify,  engage  and  challenge. 

•  For  better  data  and  data  quality  for  the  assessment  framework,  “some  interviewees  noted 
that  it  was  particularly  important  to  ask  students  to  take  the  surveys  daily  and  not  wait 
until  the  end  of  the  week  to  have  them  all  completed.”  Additionally,  the  questions 
included  the  surveys  will  need  to  be  further  evaluated  to  ensure  they  are  a  measurement 
of  the  quality  of  the  experience. 

The  USCC  will  continue  to  use  the  social  media  capabilities  and  other  communications 
capabilities  as  well  the  resources  of  the  Center  for  Internet  Security  to  include  the  Multi-State 
ISAC  community  to  increase  the  overall  numbers  of  participations  while  working  with  other 
competitions  to  broaden  the  information  and  data  for  the  assessment  framework. 

2.0  INTRODUCTION 

The  critical  infrastructure  of  this  nation  is  highly  dependent  on  cyber  infrastructure,  but  it 
does  not  enjoy  the  benefit  of  a  sufficiently  skilled  workforce  to  protect  and  guard  against  system 
failures  and  attacks.  According  to  the  Center  for  Strategic  and  International  Studies  (CSIS),  “we 
not  only  have  a  shortage  of  the  highly  technically  skilled  people  required  to  operate  and  support 
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systems  already  deployed,  but  also  an  even  more  desperate  shortage  of  people  who  can  design 
secure  systems,  write  safe  computer  code,  and  create  the  ever  more  sophisticated  tools  needed  to 
prevent,  detect,  mitigate  and  reconstitute  from  damage  due  to  system  failures  and  malicious 
acts.”1 

Experts  claim  there  are  only  1,000  people  in  the  country  currently  who  possess  the 
requisite  skills,  but  in  reality,  we  need  as  many  as  10,000  to  30, 000. 2  The  CSIS  Commission  on 
Cybersecurity  for  the  44th  Presidency  set  forth  several  recommendations  to  address  this  critical 
issue  in  its  report. 

For  the  federal  government,  the  recent  General  Accountability  Office  (GAO)  report 
references  several  Inspectors  General’s  reports  highlighting  the  need  and/or  the  difficulties  in 
filling  vacant  positions  at  the  studied  departments  and  agencies  including  a  reference  to  the 
March  2011  testimony  by  General  Keith  Alexander  that  the  military  did  not  have  enough  highly 
skilled  personnel  to  address  the  current  and  future  cyber  threats  to  your  infrastructure.3 

Technology  has  become  an  integral  part  of  our  daily  lives,  and  educators  start  introducing 
the  benefits  of  technology  as  early  as  pre-school.  As  a  result,  children  develop  technical  skills  at 
a  very  early  age.  Couple  this  access  to  technology  with  the  natural  curiosity  of  a  child  or  young 
adult  and  innocent  exploration  of  the  Internet  or  networks  (i.e.  to  “see  how  things  work”)  could 
lead  to  disaster.  According  to  a  2009  Panda  Security  study,  this  is  exactly  how  it  starts  for  a  vast 
majority  of  young  hackers.  Panda  studied  4,000  teens  from  15  to  18  years  of  age,  and  discovered 
that  “17  percent  of  adolescent  users  claim  to  have  advanced  technical  knowledge  and  are  able  to 
find  hacking  tools  on  the  Internet.  Of  these,  30  percent  claim  to  have  used  them  on  at  least  one 
occasion.  When  asked  why,  86  percent  said  that  curiosity  had  led  them  to  investigate  these 
public  tools.”4  If  left  unguided,  without  legitimate  avenues  to  test  their  skills  and  satisfy  their 
curiosity,  these  numbers  will  only  grow. 

The  above -published  statistics  regarding  the  activity  of  adolescents  and  young  adults 
show  the  need  to  intervene  at  an  early  age  and  to  channel  their  curiosity  and  technical 
proficiency  for  a  better  purpose. 

The  Center  for  Internet  Security  (CIS),  U.S.  Cyber  Challenge  (USCC),  is  well  positioned 
to  ensure  there  is  a  career  path  for  the  high-level  technical  skills  required.  USCC  believes  the 
path  must  be  available  at  an  early  age.  Finally,  the  USCC  believes  performance  metrics  should 
be  developed  in  order  to  measure  the  progress  being  made  towards  to  reduce  the  risk  for  the 
national  critical  infrastructure  regarding  cyber  threats. 

The  USCC  is  looking  for  10,000  Americans  with  the  skills  to  fill  the  ranks  for  the 
cybersecurity  practitioners,  researchers,  hunters  and  warriors.  Specifically,  USCC  objectives  are 
to: 

1.  Identify:  increase  cybersecurity  knowledge  and  talent  self-awareness  among  high  school 
students,  college  age  students,  current  professionals  and  other  interested  persons  who  are 
looking  to  enter  into  the  cyber  security  professional  ranks  and/or  further  their  existing 
careers; 


1  A  Human  Capital  Crisis  in  Cyber  security,  Center  for  Strategic  and  International  Studies,  July 

2010,  pg.  6. 

2  Id.  Pg.  1;  Cyberwarrior  Shortage  Threatens  U.S.  Security,  Tom  Gjelten,  July  19,  2010 

3  GAO,  Cybersecurity  Human  Capital:  Initiatives  Need  Better  Planning  and  Coordination, 
GAO-12-8  (Washington,  DC:  November  2011 

4  Panda  Security,  http://www.pitchengine.com/preview-release.php?id=11537 
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2.  Engage:  engage  individuals  in  order  to  increase  the  talent  pool  for  cybersecurity 
professionals,  in  both  the  public  and  private  sectors.  This  objective  includes  engaging 
individuals  across  various  demographics,  including  women  and  minorities;  and 

3.  Challenge:  provide  opportunities  for  skill  development  through  cybersecurity 
competitions  and  pathways  to  provide  increasingly  difficult  challenges  and  competitions 
as  well  as  provide  access  to  educations,  resources,  mentoring,  scholarships,  internships 
and  job  opportunities. 

To  ensure  the  IJSCC  and  other  cybersecurity  professional  development  activities  are 
meeting  and  continually  improving  on  these  objectives,  the  methodology  has  initially  been 
developed  to  provide  an  assessment  and  feedback  loop  to  serve  as  the  foundation  mechanism  for 
identifying  success  criteria,  collecting  the  necessary  data  as  well  as  establishing  the  data 
elements,  conducting  appropriate  collection  activities  and  analysis,  and  addressing  the  challenges 
through  the  short;  mid  and  long  term  of  our  efforts. 

3.0  METHODS,  ASSUMPTIONS,  AND  PROCEDURES 

3.1  Assumptions 

In  order  to  provide  context  for  the  initial  assessment  framework  developed  under  this 
research  proposal,  there  are  a  number  of  important  assumptions.  These  assumptions  are  as 
follows: 

•  A  more  cybersecurity  aware  populous  will  mitigate  against  the  Nation’s  cyber  risks  and 
improve  the  Nation’s  cybersecurity  posture. 

•  Individuals  with  the  right  skills  and  education  in  cybersecurity,  in  the  appropriate  federal 
and  industry  roles  and  positions,  will  improve  the  cyber  security  posture  of  the 
Nation/National  critical  information  technology  (IT)  infrastructure. 

Given  these  assumptions,  the  initial  assessment  framework  does  not  focus  on  assessing  or 
quantifying  the  Nation’s  cyber  risk  profile  or  cyber  risk  posture  as  this  is  done  at  the  national 
level  and  outlined  in  reports  such  as  the  recently  released  report  by  the  Office  of  the  National 
Counterintelligence  Executive.5  This  initial  assessment  framework  is  designed  to  assess  and 
improve  the  activities  of  the  USCC  and  the  cybersecurity  professional  activities  in-line  with  the 
objectives  of  identify,  engage  and  challenge. 

Furthermore,  the  initial  assessment  framework  is  designed  to  support  the  underlying 
USCC’s  “pathway”  construct.  In  this  construct,  the  field  of  cybersecurity  is  viewed  as  a  pathway 
with  multiple  entry  and  exit  points,  as  well  as,  various  paths  to  differing  destinations  for 
education  and  eventual  job  entry  or  if  you  are  already  in  the  job  market,  re-entry  for 
developmental  purposes  to  become  more  cyber-enhanced  and/or  a  cybersecurity  professional. 
This  analogy  allows  the  USCC  to  represent  that  many  may  be  entering  the  field  at  different 
points  of  time  in  their  career,  they  may  be  coming  from  differing  locations  or  career 
backgrounds,  and  ultimately,  they  may  choose  differing  paths  or  specialties  within  the 
cybersecurity  field.  In  contrast,  the  “pipeline”  analogy  is  unfitting  the  USCC  approach  because 
it  implies  one  single  point  of  entry  and  highly  structure,  linear  path  to  a  single  destination.  As 


5  “Foreign  Spies  Stealing  US  Economic  Secrets  in  Cyberspace:  Report  to  Congress  in  Foreign 
Economic  Espionage  2009-2011,”  October  201 1 
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such,  the  initial  assessment  framework  looks  at  the  information  on  a  participant’s  background, 
experience,  skills,  and  successes  to  help  assist  and  map  a  pathway  through  to  their  choice. 

3.2  Methods 

The  initial  assessment  framework  is  geared  towards  measuring  the  success  of  the  USCC 
and  cybersecurity  professional  development  activities  in  achieving  the  USCC’s  objectives. 

These  successes  and  associated  challenges  are  addressed  over  three  distinct  time  horizons;  short¬ 
term,  mid-term;  and  long-term.  Each  of  these  time  horizons  presents  a  distinct  set  of  challenges, 
relevant  data  and  questions,  as  well  as,  a  discrete  set  of  tools  for  conducting  data  collection  and 
facilitating  assessment  activities.  Although  the  specific  data  elements  and  collection  tools  will 
change  over  the  varying  time  horizons,  a  number  of  underlying  questions  being  addressed  by  the 
assessment  framework  remain  the  same.  These  underlying  questions  are  as  follows: 

•  Does  participation  in  structured  cyber  security  activities  increase  individual  self- 
awareness  of  talent  and /  or  interest  in  cyber  security  disciplines  and  issues? 

•  What  activities  help  individuals  identify  their  talent  in  cyber  security  and  which  with 
subsequently  encourage  participation? 

•  What  cyber  security  activities  provide  the  most  incentive  to  continue  on  the  pathway? 

•  What  incentives  (e.g.,  prize  money,  prize  software,  scholarships,  etc.)  provide  the  most 
incentive  to  continue  on  the  pathway? 

•  Does  participation  in  cyber  security  activities  at  the  high  school  and  college  level 
encourage  a  career  in  cyber  security? 

•  Does  a  structured  and  well-articulated  path  to  a  career  in  cyber  security  increase 
considerations  for  a  career  in  cyber  security? 

•  Does  a  ‘pathway’  using  discrete  levels  of  achievement  that  include  possible  college 
admission,  access  to  financial  aid,  and /  or  employment  opportunities  create  a  strong 
motivator  for  participation? 

In  addition  to  these  underlying  questions  the  assessment  framework  addresses  three 
overarching  focus  areas  as  follows: 

Adoption/ Participation  -  This  focus  area  specifically  addresses  the  degree  to  which  individuals 
are  participating  and  continue  to  participate  in  USCC  and  related  activities. 

Usage  -  This  focus  area  specifically  addresses  participant  use  of  the  various  USCC  tools  and 
systems. 

Customer  Satisfaction  -  This  focus  area  specifically  addresses  the  level  of  benefit,  knowledge, 
enjoyment  and  value  an  individual  takes  from  USCC  and  related  activities. 

In  order  to  answer  these  underlying  questions,  address  the  overarching  focus  areas,  and 
validate  the  USCC  objectives,  there  must  be  quantifiable  mechanisms  to  enable  the  capture  of 
data  across  each  time  horizon.  Feedback  from  the  analysis  and  study  of  this  data  will  enable 
USCC  to  validate  the  success  or  failure  of  the  objectives,  as  well  as,  provide  data  to  better  enable 
adjustments  to  better  align  to  USCC  Objectives. 

3.2.1  Short-Term  Methodology.  The  objectives  of  the  assessment  framework  in  short-term 
are  geared  toward  identifying  the  key  data  elements,  deriving  assessment  baselines  and 
establishing  the  processes  by  which  data  will  be  collected  over  the  mid  and  long  terms. 

The  table  below  provides  an  overview  of  the  key  objectives,  data  and  collection  methods 
for  each  of  the  overarching  focus  areas  in  the  short-term  as  outlined  in  Table  1,  below: 
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Table  1,  Short-Term  Methodology 


Adoption/  Participation 

Usage 

Customer  Satisfaction 

Questions/ 

Objectives 

•  Are  individuals  engaging 
USCC  and  related  events? 

•  What  capabilities/tools  are 
participants  using? 

•  What  are  the  capabilities 
and  requirements  for 
inclusion  as  a  USCC 
recognized  system,  activity, 
initiative? 

•  Are  participants  interested  in 
continued  participation? 

•  Do  participants  perceive  value 
from  participating? 

•  Do  participants  gain 
knowledge  from 
participating? 

Data 

•Number  of  Registrants 
•Number  of  Participants 

•  Participant  Gender 

•  Participant  Ethnicity 

•  Participant  Age 

•  Participant  GPA?  Education 

•  Participant  Career  Interests 

•  Participant  Proficiencies 

•  Participant  Extracurricular 

O  Clubs,  Sports,  etc. 

•  Tool,  feature  and  usage 
analysis 

•  Participant  likelihood  of  future 
participation 

•  Participant  assessment  of 

O  Event  value 

O  Event  effectiveness 

O  Knowledge  acquired 

Proposed 

Metrics 

•  %  of  new  users  attending 
USCC  events 

•  %  of  returning  users  attending 
USCC  events 

•  #  of  High  Schools 
participating  in  USCC 
activities 

•  #  of  Universities  participating 
in  USCC  activities 

•  #  of  women  attending  USCC 
events 

•  #  of  minorities  attending 

USCC  events 

•  %  of  monthly  unique  visitors 
to  the  USCC  website 

•  %  of  users  downloading  free 
web  tutorials 

•  %  of  users  using  toolkits 

•  #  of  users  participating  in 
USCC  web  games 

•  #  of  users  participating  in 
‘in-person’  competition 

•  %  of  users  who  want  to 
continue  with  other  USCC 
activities 

•  %  of  users  who  feel  they 
gained  knowledge 

•  %  of  users  who  would 
recommend  USCC  activities 
to  others 

Collection 

Tools 

•  Registration  forms 

•  Participant  profiles 

O  Ex:  Facebook,  Twitter 

•  N/A 

•  Surveys/  Interviews 

•  Blogs 

•  Pre  and  post  event  knowledge 
assessments 

Adoption/  Participation 

In  the  short-term,  adoption/  participation  is  predominantly  about  assessing  how  many 
people  are  engaging  IJSCC  events  and  activities  and  developing  an  initial  understanding  of  the 
characteristics  of  these  participants.  Data  should  be  collected  and  analyzed  to  develop  a  baseline 
of  participation.  This  baseline  can  then  be  used  in  subsequent  analysis  to  assess  the  effectiveness 
of  changes  to  the  activities  as  well  as  identify  demographics  for  targeted  engagement,  such  as 
women  or  individuals  with  traditionally  no  cyber  security  associated  interests. 
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To  augment  the  initial  data  collected  from  USCC  events  in  the  short-term,  identifying  and 
incorporating  structured  data  from  schools,  universities  and  other  data  aggregation  groups  will  be 
effective  in  developing  a  more  in-depth  understanding,  as  well  as,  developing  more 
comprehensive  baselines  for  future  assessments.  This  structured  information  may  include  student 
enrollment,  graduation  and  success  rates  as  well  as  employment  and  trends. 

Usage 

In  the  short-term  much  of  the  data  collection  and  feedback  mechanisms  will  be  reliant  on 
existing  ubiquitous  technologies  such  as  Facebook,  Twitter,  blogs,  survey  tools  and  others. 
Although  these  tools  lack  the  sophisticated  analytic  capabilities  necessary  in  the  long-term,  they 
provide  an  accelerated  avenue  for  the  immediate  collection  of  data.  As  such,  usage  in  the  short¬ 
term  is  an  assessment  of  the  collection  mechanisms  to  develop  requirements  for  a  USCC 
sponsored  system  which  is  added  to  the  framework  in  the  mid-term  time  horizon. 

Customer  Satisfaction 

Short-term  customer  satisfaction  is  essentially  assessing  ‘Are  participants  interested  in 
continued  participation?’  as  well  as  developing  a  baseline  of  satisfaction.  Similar  to  the  adoption/ 
participation  baseline,  the  satisfaction  baseline  serves  as  a  preliminary  point  for  assessing  the 
effectiveness  of  changes  to  the  activities  and  events.  This  baseline  will  describe  how  effective 
and  successful  participants  judge  USCC  events  to  be. 

However,  the  most  critical  component  of  the  short-term  is  assessing  each  participant 
interested  in  future  participation.  Since  collecting  information  on  participants  and  providing 
linkages  and  analysis  to  the  USCC  Objectives  requires  access  to  significant  amounts  of  data  over 
time-  it  is  critical  to  ensure  interest  in  future  participation.  If  participants  are  not  interested  in 
continuing  their  participation  then  there  is  no  need  to  conduct  assessments  in  later  phases  and  the 
ability  to  track  improvements  and  long  terms  results  will  be  compromised. 

3.2.2  Mid-Term  Methodology.  The  objectives  of  the  initial  assessment  framework  in  mid¬ 
term  are  geared  toward  developing  the  feedback  loop  to  implement  and  assess  improvements  to 
the  process.  Moreover,  the  mid-term  is  the  transition  point  to  develop  a  more  robust 
infrastructure  for  data  collection,  user  management  and  the  development  of  a  cybersecurity 
social  network.  The  mid-term  effort  is  collectively  a  transition  and  refining  period  to  establish 
the  “in-house”  and  on-going  infrastructure  to  support  this  effort’s  long  term  objectives  as  well  as 
to  refine  the  baseline,  questions  and  assessment  mechanisms. 

The  Table  2  below  provides  an  overview  of  the  key  objectives,  data  and  collection 
methods  for  each  of  the  overarching  focus  areas  in  the  mid-term. 


Table  2,  Mid-Term  Methodology 


Adoption/  Participation 

Usage 

Customer  Satisfaction 

Questions/ 

Objectives 

•  Is  the  baseline  established  in 
the  short-term  effective? 

•  Is  participation  limited  to 
specific  participants  (age, 
gender,  interests,  etc.)? 

•  Are  participants  continuing  to 
enter  the  ‘highway’?  If  so,  at 
what  entry  points? 

•  Does  the  USCC  sponsored 
platform  incorporate  the 
necessary  features  and 
capabilities? 

•  Is  there  an  effective  interface 
for  the  USCC  sponsored 
platform? 

•  Are  individuals  previously  not 
engaged  in  cyber  security 
discovering  interest  in  the 
field? 

•  Do  users  like  the  tool? 

•  Are  users  retaining  skills/ 
knowledge? 

•  Are  users  incorporating  USCC 
knowledge  in  their  daily 
activities? 
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Data 

•  Short-term  baseline  gap 
analysis 

•  Short-term  demographic 
representation  analysis 

•  Beta  test  participant  group 
feedback 

•  Initial  deployment  user 
feedback 

•  Participant  experience  stories 
(blog) 

•  Focused  participant  capability 
and  skill  assessment 

•  Participant  habits  and 
awareness 

Proposed 

Metrics 

•  %  of  new  users  attending 
USCC  events 

•  %  of  returning  users  attending 
USCC  events 

•#  of  High  Schools 
participating  in  USCC 
activities 

•#  of  Universities  participating 
in  USCC  activities 

•  #  of  other  Cyber  competitions 
promoting  USCC  events 

•  %  of  monthly  unique  visitors 
to  the  USCC  website 

•  %  of  users  downloading  free 
web  tutorials 

•  %  of  users  using  toolkits 

•  #  of  users  participating  in 
USCC  web  games 

•  #  of  users  participating  in  ‘in- 
person’  competition 

•  #  of  users  signed  up  for  USCC 
sponsored  platform 

•  #  of  users  actively  using  the 
USCC  sponsored  site 

•  %  of  users  who  want  to 
continue  with  other  USCC 
activities 

•  %  of  users  who  feel  they 
gained  knowledge 

•  %  of  users  who  would 
recommend  USCC  activities 
to  others 

•  %  satisfied  with  USCC 
sponsored  site 

Collection 

Tools 

•  Participant/  Registrant  Rates 

•  Structured  testing  feedback 
process 

•  User  interviews  and  surveys 

•  Periodic  proficiency  and 
retention  assessments 

•  Habit  and  practices  (activity) 
assessments 

Adoption/  Participation 

In  the  mid-term,  adoption/  participation  is  focused  on  refining  the  participation  baseline 
and  assessing  underrepresented  demographics.  Before  changing  the  adoption/  participation 
baseline,  a  gap  analysis  would  be  conducted  on  the  data  elements  being  captured  from  the  short¬ 
term.  Areas  or  elements  of  information  which  are  not  collected  or  which  are  collected  but  at  an 
unusable  quality  can  be  revised  to  ensure  comprehensiveness  and  completeness.  Following  each 
update  to  the  framework,  the  modified  results  would  be  compared  to  the  short-term  baseline 
results  over  a  3,  6,  12  month  window  to  ensure  the  effectiveness  of  each  modification. 

Additionally  in  the  mid-term,  analysis  on  key  demographics  can  provide  insights  about 
who  is  and  who  is  not  participating.  This  can  be  used  to  both  refine  data  collection  efforts  and  to 
inform  targeted  demographic  engagement  efforts  in  the  long-term. 

Lastly,  participation  in  the  mid-term  includes  continued  analysis  of  new  participants. 
Because  IJSCC  has  adopted  the  ‘pathway’  model  there  is  the  expectation  that  participants  will 
continually  be  entering  and/or  re-entering  and  the  data  elements  identified  in  the  short-term  will 
continue  to  be  collected  from  these  individuals  as  they  enter  the  cybersecurity  pathway. 

Usage 

Although  systems  like  Facebook,  Twitter  and  other  social  media  outlets  provide  a 
tremendous  starting  point,  the  long  term  needs  of  this  project  require  more  administrative  access 
to  the  underlying  framework.  Specifically,  the  ability  to  tailor  user  profiles,  manage  user 
interactions  and  control  specific  data  elements  for  each  user  to  maximize  the  effectiveness  and 
efficiency  of  data  collection  and  analysis.  The  mid-term  usage  focus  is  on  the  transition  from 
quick  collection  tools  to  a  more  robust,  USCC  sponsored  platform  which  the  USCC  calls  the 
“community  gym”  or  the  “teaching  hospital”  depending  on  the  skills  levels  and  participation  of 
the  individual  competitor. 
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This  USCC  sponsored  platform  will  provide  the  participant  management,  event 
registration,  event  engagement  and  social  network  components  for  the  USCC  participant 
community.  In  the  mid-term,  the  utility  will  be  monitored  with  feedback,  provided  by  focus 
groups,  about  the  interface,  capabilities,  usability,  user  experience  and  overall  effectiveness  of 
the  USCC  sponsored  platform.  Additionally,  as  the  platform  is  finalized  and  moved  into 
production  during  the  mid-term,  usage  will  incorporate  user  feedback  and  provide  a  structured 
requirements  management  process. 

Customer  Satisfaction 

In  the  mid-term,  customer  satisfaction  is  focused  on  assessing  participant’s  discovery  of 
previously  unknown  cyber  security  talent  or  interest  as  well  as  participant’s  retention  and 
incorporation  of  skills  and  knowledge  into  their  daily  lives.  Assessing  a  success  rate  for  helping 
individuals  discover  a  previously  unknown  interest  or  ability  in  the  cyber  security  field  directly 
informs  one  of  the  objectives.  Moreover,  if  by  the  mid-term  this  is  not  effectively  happening,  a 
deeper  assessment  of  demographics  and  participant  experience  would  be  evaluated  to  ensure  the 
proper  breadth  of  individuals  is  being  engaged  in  USCC  activities. 

Additionally  in  the  mid-term,  the  framework  is  looking  to  assess  participant’s 
implementation  and  retention  of  skills  and  knowledge  gained  as  a  result  of  USCC  activities. 
Collecting  and  assessing  retention  will  help  in  the  modification  of  USCC  material  and  events  to 
better  facilitate  the  delivery  of  core  concepts  to  participants.  This  will  also  provide  insight  into 
which  elements  of  USCC  events  are  most  effective  in  immediately  impacting  participant  habits. 
3.2.3  Long-Term  Methodology.  The  objectives  of  the  assessment  framework  in  the  long-term 
are  geared  toward  the  deployment  of  sponsored  collection  platforms,  continual  refinement  of 
data  collection  and  analysis,  and  participant  tracking. 

Table  3  below  provides  an  overview  of  the  key  objectives,  and  data  and  collection 
methods  for  each  of  the  overarching  focus  areas  in  the  long-term. 


Table  3,  Long-Term  Methodology 


Adoption/  Participation 

Usage 

Customer  Satisfaction 

Questions/ 

Objectives 

•  Are  participants  continuing  to 
careers  in  the  field? 

•  Are  users  continuing 
participation  in  USCC 
activities? 

•  Engage  underrepresented 
demographics. 

•  Are  underrepresented 
demographics  participating  in 
USCC  activities? 

•  Are  participants  continuing  to 
enter  the  ‘highway’? 

•  Is  the  data  entered  in  the 

USCC  sponsored  platform  of 
sufficient  quality  and 
completeness 

•  Do  the  features  and 
capabilities  of  the  system 
effectively  encourage  and 
support  the  USCC  cyber 
security  community? 

•  Are  individuals  previously  not 
engage  in  cyber  security 
discovering  interest  in  the 
field? 

•  Are  users  retaining  skills/ 
knowledge? 

•  Are  users  incorporating  USCC 
knowledge  in  their  daily 
activities? 

Data 

•  Participant  career  information 

•  Participant  activity  in  USCC 
and  associated  activities 

•  Marketing  and  outreach 
analysis 

•  Platform  data  analysis 

•User  feedback 

•  Participant  experience  stories 
(blog) 

•  Focused  participant  capability 
and  skill  assessment 

•  Participant  habits  and 
awareness 
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Proposed 

Metrics 

•  %  of  new  users  attending 
USCC  events 

•  %  of  returning  users  attending 
USCC  events 

•  #  of  High  Schools 
participating  in  USCC 
activities 

•  #  of  Universities  participating 
in  USCC  activities 

•  #  of  users  obtaining  cyber 
security  jobs  in  the  civilian 
sector 

•  #  of  users  obtaining  cyber 
security  jobs  in  the  federal 
sector 

•  %  of  monthly  unique  visitors 
to  the  USCC  website 

•  %  of  users  downloading  free 
web  tutorials 

•  %  of  users  using  toolkits 

•  #  of  users  participating  in 
USCC  web  games 

•  #  of  users  participating  in  ‘in- 
person’  competition 

•  #  of  users  signed  up  for  USCC 
sponsored  Platform 

•  #  of  users  actively  using  the 
USCC  sponsored  site 

•  %  of  users  who  want  to 
continue  with  other  USCC 
activities 

•  %  of  users  who  feel  they 
gained  knowledge 

•  %  of  users  who  would 
recommend  USCC  activities 
to  others 

•  %  satisfied  with  USCC 
Sponsored  site 

•  %  satisfied  with  their  Cyber 
Career 

Collection 

Tools 

•  Mango  and  other  USCC 
sponsored  participant 
platforms 

•  Surveys/  Interviews 

•  Periodic  proficiency  and 
retention  assessments 

•  Habit  and  practices  (activity) 
assessments 

Adoption/  Participation 

In  the  long-term,  adoption/  participation  is  concerned  with  the  continued  entry  of 
individuals  into  the  USCC  cybersecurity  pathway,  encouraging  underrepresented  demographics 
to  enter  the  pathway  as  well  as  continued  participation  in  USCC  events  and  the  transition  of 
individuals  from  USCC  to  cybersecurity  careers.  Utilizing  the  USCC  sponsored  platform 
developed  in  the  mid-term,  trends  and  analysis  can  be  conducted  following  individuals  through 
and  following  their  USCC  engagement. 

Usage 

Usage  in  this  time  horizon  is  focused  on  the  implementation  and  effectiveness  of  the 
USCC  sponsored  platform.  In  particular,  long-term  will  incorporate  the  continual  assessment  of 
the  data,  features  and  capacities  within  the  “community  gym,”  as  well  as,  provide  the  process  for 
the  continual  improvement  and  feedback  of  the  system. 

Customer  Satisfaction 

In  the  long-term,  customer  satisfaction  is  the  continuation  of  participants  skill  and 
knowledge  retention  and  application. 

3.3  Procedures 

The  initial  assessment  framework  is  designed  to  address  the  three  time  horizons  as 
outlined  above.  The  following  procedures  discussed  are  addressing  the  work  completed  within 
the  period  of  performance  of  this  effort  which  is  specifically  focused  on  the  short-term  horizon. 
Specifically,  the  data  collection  took  place  prior  to  a  planned  event,  during  the  event  and 
following  an  event.  Events  measured  by  the  USCC  will  include  competitions,  trainings,  camps 
and  other  structured  cybersecurity  professional  development  activities.  Additionally,  the  USCC 
employed  the  expertise  of  the  team  from  Louisiana  Tech  University  to  validate  the  data 
collection  procedures  and  the  initial  assessment  framework  in  order  to  ensure  performance 
metrics  and  activities  would  address  and/or  highlight  potential  gaps  to  address  going  forward. 
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3.3.1  Pre-Event.  The  data  collected  in  this  time  period  revolves  around  participant 
demographics,  professional  or  educational  background  and  personal  interest  in  the  material.  One 
objective  during  the  pre-event  collection  period  is  to  assess  participant  expectations  and 
perceived  value.  Participant  expectation  is  a  measure  of  a  participant’s  anticipation  of  the  quality 
and  rigor  of  each  event.  This  includes  elements  of  both  an  individual’s  personal  experience  with 
other  similar  events,  as  well  as  an  assessment  of  each  individual’s  evaluation  of  the  reputation  of 
the  event.  Within  the  USCC  framework,  perceived  value  represents  an  assessment  of  the 
perceived  or  expected  benefit  prior  to  a  participant’s  engagement  in  the  activity,  the  value  (trade 
off  of  participant’s  time  investment  versus  perceived  benefit),  and  the  event’s  objectives  (the 
applicability  of  the  specific  event  to  the  perceived  needs  of  the  participant). 

3.3.2  In-Event.  The  data  collected  in  this  time  period  revolves  around  participant  engagement, 
capability  to  process  information  and  participate  in  event  activities,  and  each  participant’s 
enjoyment  of  the  material.  One  objective  during  this  period  is  to  assess  the  rigor,  quality  and 
understandability  of  the  materials  and  activities  incorporated  into  the  event  as  well  as  assessing 
participant  satisfaction  through  participant  input  and  feedback  (both  positive  and  negative). 
Within  the  USCC  framework,  satisfaction  is  a  measure  of  each  participant’s  level  of  personal 
benefit,  knowledge  acquisition,  and  the  enjoyment  an  individual  takes  from  USCC  and  related 
activities.  Feedback  includes  an  assessment  of  a  participant’s  expectations  (perceived  value) 
versus  activity  outcomes  (satisfaction)  as  well  as  indicators  from  the  individual  on  how  these  two 
elements  differ.  Furthermore,  developing  and  implementing  data  collection  during  this  time 
period  also  provides  for  feedback  specific  to  individual  components  of  the  event  and  prevents 
inadvertent  averaging  of  feedback  in  a  summary  collection  after  the  event. 

3.3.3  Post-Event.  The  data  collected  in  this  time  period  revolves  heavily  around  participant 
satisfaction,  knowledge  acquisition  and  retention,  and  continued  engagement  in  the  field  of  cyber 
security.  One  objective  during  this  period  is  to  assess  adoption  and  participation  as  well  as  usage 
and  application  of  cyber  security  skills,  capabilities  and  techniques.  Adoption  and  participation 
are  similar  to  the  concept  of  customer  loyalty.  This  includes  an  assessment  of  a  participant’s 
likelihood  to  incorporate  USCC  elements  into  their  daily  lives,  to  pursue  a  career  in  USCC 
related  disciplines,  and  to  continue  participating  in  future  security  and  USCC  events. 
Furthermore,  another  objective  during  this  period  is  assessing  usage  and  application,  which  is 
focused  on  assessing  each  participant’s  use  of  USCC  provided  toolkits,  systems,  infrastructure 
and  connections. 

Appendix  A,  “Assessment  Techniques,”  provides  a  brief  overview  of  the  types  of 
activities  to  take  place  within  each  time  horizon  at  each  proposed  assessment  period.  The 
Appendix  also  includes  initial  specific  objectives,  questions,  data,  methodologies,  technologies 
utilized  or  to  be  utilized  within  each  time  horizon. 

Appendix  B,  “Assessment  Database  Structure,”  provides  an  underlying  database  structure 
of  participant  and  survey  data  to  facilitate  data  collection  and  reporting.  In  the  short-term  time 
horizon,  the  database  will  be  a  subset  of  these  elements  leveraging  a  third-party  tool 
(SurveyMonkey)  for  the  short-term  effort.  In  the  mid  and  long  terms,  USCC  will  migrate  toward 
a  customized  participant  management  platform  that  will  allow  users  and  instructors  to  maintain 
profiles,  participants  to  register  for  competitions,  to  deliver  data  to  all  USCC  stakeholders,  and  to 
facilitate  surveys  and  feedback.  The  long-term  section  discusses  the  database  structure  for 
maintaining  data  within  that  platform. 
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4.0  RESULTS  AND  DISCUSSION 


In  July  and  August  of  201 1,  the  USCC  hosted  cyber  security  camps  in  Virginia, 
Maryland,  Missouri,  California,  and  Delaware.  These  invitation  only  camps  provided  high 
school,  college  and  young  professionals  with  one  week  of  specialized  cyber  security  training 
presented  by  college  faculty,  cyber  security  experts  and  included  activities  such  as  a  job  fair, 
ethics  workshop,  a  capture-the-flag  competition,  industry  roundtable  and  an  awards  ceremony  on 
the  last  day.  The  Table  4  below  provides  an  overview  of  the  2011  camps. 


Table  4,  Summary  of  Summer  Camp  2011  Information 


State 

Dates 

Location 

Information 

California 

July  1 1  - 
July  15 

California  State  Polytechnic  University,  Pomona 
3801  West  Temple  Avenue 

Pomona,  CA  91768 

Overnight  camp  open  to 
eligible  participants  from 
CA,  AZ,  NV,  OR 

Maryland 

July  1 1  - 
July  15 

Community  College  of  Baltimore  County,  Essex 
7201  Rossville  Boulevard 

Baltimore,  MD  21237 

Day  camp  open  to  high 
school  students  only 
from  this  state 

Missouri 

July  25  - 
July  29 

University  of  Missouri,  Columbia 

Jesse  Hall 

Columbia,  MO  65201 

Overnight  camp  open  to 
eligible  participants  from 
all  states  west  of  the 
Mississippi  (except  those 
attending  the  California 
Camp) 

Virginia 

August  1  - 
August  5 

J.  Sargeant  Reynolds  Community  College, 
Richmond 

Parham  Road  Campus 

1651  East  Parham  Road 

Richmond,  Virginia,  23228 

Overnight  camp  open  to 
eligible  participants  from 
all  states  east  of  the 
Mississippi  (except 
Delaware) 

Delaware 

August  8  - 
August  12 

Delaware  Technical  and  Community  College, 

Dover 

Terry  Campus 

100  Campus  Drive 

Dover,  DE  19904 

Day  camp  open  to 
eligible  participants  from 
Delaware 

The  Cyber  Camps  provide  crucial  skills  development  and  enable  USCC  to  tap  into  the 
tremendous  talent  across  our  nation  to  identify  those  with  a  passion  for  security  and  a  desire  to 
put  their  skills  to  good  use  in  addressing  our  Nation’s  cyber  security  workforce  challenges.  In 
addition  to  providing  expert  training  for  participants  to  improve  their  skills  and  marketability,  the 
Cyber  Camps  provided  attendees  the  opportunity  to  engage  with  major  technology  companies 
and  government  agencies  at  onsite  job  fairs  for  scholarship,  internship  and  employment 
opportunities  as  well  as  engage  industry  professionals  in  an  ethics  panel. 

The  2011  camps  are  "invitation  only"  after  students  initially  completed  the  on-line 
competition,  Cyber  Quest.  Furthermore,  the  camps  were  provided  as  either  day  camps  or  in¬ 
residence  overnight  camps.  For  the  overnight  camps,  there  was  a  minimum  age  requirement  of 
1 8  years  and  older. 

During  the  2011  camps,  USCC  utilized  an  automated  survey  tool,  SurveyMonkey,  to 
capture  participant  demographic  and  event  feedback  information  from  each  event.  In  the  2010 
pilot  camps  this  data  collection  was  done  with  hardcopy  surveys. 
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This  report  provides  an  overview  of  the  data  collection  methods  in  both  the  2011  camps 
and  the  2010  pilot  camps,  an  analysis  of  the  data  collected  in  each  year,  a  comparison  of  the  data 
from  the  2010  pilot  and  2011  camp  surveys,  a  summary  of  findings  from  interviews  with  camp 
instructors  and  administrators,  a  summary  of  overarching  observations  derived  from  the  data, 
analysis  and  interviews  as  well  as  next  steps  for  camps  in  2012. 

4.1  2011  Camp  Administrator  Interview  Summary 

Following  the  completion  of  the  2011  Camps,  a  number  of  key  individuals  involved  in 
the  administration,  execution  and  setup  of  the  Camps  were  interviewed  to  capture  their  thoughts, 
experiences,  concerns,  issues  and  recommendations.  The  Table  5  below  summarizes  the 
individuals  interviewed: 


Table  5,  2011  Camp  Administrator  Call  Summary 


Interview 

Date/  Time 

Individuals  in  Attendance 

Missouri  Camp  Debrief 

September  1,  2011  -  10:30a 

Beth  Fisher,  Karen  Evans,  Renee  Mclaughlin, 
Patrick  Ansaldi,  Douglas  C  Houghton  l 

Maryland  Camp  Debrief 

September  1,  2011  -  11:30a 

Casey  O’Brien,  Davina  Pruitt,  Randy 
Marchany,  Karen  Evans,  Renee  Mclaughlin, 
Patrick  Ansaldi,  Douglas  C  Houghton 

Delaware  Camp  Debrief 

September  2,  201 1  -  10:30a 

Elaine  Starkey,  Chase  Cotton,  Jared  Bates, 
Karen  Evans,  Renee  Mclaughlin,  Patrick 
Ansaldi,  Douglas  C  Houghton 

California  Camp  Debrief 

September  2,  201 1  -  5:00p 

Dan  Manson,  Anna  Carlin,  Karen  Evans, 

Renee  Mclaughlin,  Patrick  Ansaldi,  Douglas 

C  Houghton 

General  Camp  Debrief 

September  6,  201 1  -  10:30a 

Rudy  Pamintuan,  Karen  Evans,  Patrick 

Ansaldi,  Douglas  C  Houghton 

Virginia  Camp  Debrief 

September  6,  201 1  -  12:30p 

Kristopher  Cox,  Karen  Evans,  Renee 
Mclaughlin,  Patrick  Ansaldi,  Douglas  C 
Houghton 

General  Camp  Debrief 

September  23,  201 1  -  3:00p 

Sonny  Sandelius,  Douglas  C  Houghton 

General  Camp  Debrief 

September  26,  201 1  -  4:00p 

Randy  Marchany,  Patrick  Ansaldi,  Douglas  C 
Houghton 

General  Camp  Debrief 

September  30,  201 1  -  10:30a 

Karen  Evans,  Patrick  Ansaldi,  Douglas  C 
Houghton 

4.1.1  Common  Interview  Findings.  Across  all  the  interviews  a  number  of  common  themes 
were  captured  regarding  the  2011  Cyber  Camps.  The  list  below  explains  each  of  these  common 
elements.  Listed  numerically  for  ease  of  reference,  but  they  are  not  in  a  particular  order. 
Appendix  C,  “2011  Competition  Data  Collection  Analysis,”  includes  the  survey  structures  and 
the  data  collection  results  from  the  camp  participants. 

1.  Logistics.  Nearly  all  of  the  interviewees  acknowledged  that  logistics  played  a  significant 
role  in  the  success  or  complication  of  their  particular  camp.  The  two  primary  logistical 
difficulties  were  transportation  from  the  nearest  airport  to  the  camp  and  from  the 
overnight  residency  to  the  daily  camp  location.  Camps  who’s  location  was  a  significant 
distance  from  the  nearest  airport,  in  excess  an  hour,  noted 
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2.  Facilities.  Nearly  every  interviewee  noted  some  issue  with  the  setup  or  layout  of  the 
facility.  Common  issues  included  the  lack  of  adequate  power  outlets  for  student  laptops, 
the  lack  of  network  bandwidth  for  student  connectivity  and  the  lack  of  tabletop  space  for 
students  to  work.  Some  camps  also  noted  complications  with  the  rooms,  specifically  that 
auditoriums  worked  well  for  lectures  but  failed  to  provide  space  for  hands-on  and  team 
activities.  Alternatively,  camps  that  used  tables  noted  that  small  tables  (8  people)  worked 
well  for  team  activities  but  presented  complications  during  lectures  since  it  was  easy  for 
students  to  get  distracted. 

3.  Teaching  Assistants  (TAs).  Many  of  the  participants  sighted  the  quality  of  TAs  as  either 
a  major  contributor  to  the  success  of  the  camp  or  as  a  major  problem  at  the  camp.  From 
the  interviews  it  appears  that  TAs  who  underwent  more  vetting  scrutiny,  had  more 
training,  and  were  provided  materials  outlining  their  specific  roles  and  responsibilities 
provided  significantly  more  impact  and  benefit  to  the  quality  of  the  camp.  Alternatively, 
in  some  cases  those  without  structured  guidance  or  vetting  became  a  distraction  to  the 
students  and  a  disruption  to  the  camp. 

4.  Career  fair  expectations.  Across  the  different  camps  the  job  fair  had  different  setups 
and  contained  different  elements.  Some  fairs  provided  resume  development,  career  panels 
to  discuss  the  field,  rotating  speed-date  style  interviews  with  corporate  representatives, 
and  some  incorporated  professional  associations  and  government  representatives 
alongside  hiring  companies.  Generally  all  the  interviewees  agreed  the  career  fair  needed 
to  be  given  more  time,  that  students  generally  considered  the  career  fair  a  central 
component  of  the  camp,  and  that  a  structured  resume  development  element  would  highly 
benefit  the  students  and  attending  companies.  In  some  camps,  the  administrator  collected 
resumes  in  advance  and  provided  them  to  attending  companies  which  received  good 
feedback  from  the  attending  companies.  Some  interviewees  did  not  difficulty  in 
identifying  companies  to  attend  the  career  fair,  particularly  if  the  scheduling  and 
coordination  of  those  companies  didn’t  begin  very  early. 

5.  Skill  Granularity.  The  mix  of  students  with  different  skill  levels  within  the  camp  raised 
both  positive  and  negative  feedback.  The  suggestion  to  divide  camps  into  introduction, 
intermediate  and  expert  level  courses  was  equally  mixed  in  response.  Many  felt  the 
students  benefited  from  engaging  other  students  with  more  or  less  skill,  allowing  them  to 
learn  from  them  and /  or  teach  them.  On  the  other  hand,  interviewees  noted  the  more 
proficient  students  tended  to  lose  interest  in  the  introductory  material. 

6.  Camp  preparation  manual.  Many  of  the  interviewees  expressed  the  desire  for  a 
comprehensive  camp  administrator’s  manual.  There  was  a  general  sense  that  many  of  the 
camp  materials,  setup  issues,  and  logistic  considerations  could  have  been  brought  to  the 
administrator’s  attention  and  resolved  had  they  had  a  manual.  Furthermore,  a  number  of 
the  interviewees  noted  a  disconnection  between  USCC,  camp  administrator  and  host 
facility  roles  and  expectations.  Most  felt  a  manual  which  clearly  outlined  expectations  of 
all  the  involved  organizations  would  be  greatly  beneficial. 

7.  Read  ahead  materials.  Nearly  all  the  interviewees  agreed  that  more  informational 
material  provided  to  both  students  and  TAs  prior  to  the  camp  would  be  helpful. 
Information  for  students  on  the  logistics  of  the  camp  and  the  area,  the  layout  of  the  host 
campus,  meal  plans,  local  transportation,  and  daily  course  outlines  would  help  ensure 
they  arrive  informed  and  reduce  time  lost  to  coordinating  students.  Information  for  TAs 
on  their  role,  responsibilities,  acceptable  and  unacceptable  behavior,  daily  schedules,  key 
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personnel  and  contact  information  would  help  ensure  TAs  better  understand  their  role 
within  the  camps. 

8.  Camp  location  applications.  In  addition  to  the  manual,  a  number  of  the  interviewees 
suggested  an  application  for  camp  locations  to  ensure  the  host  had  the  appropriate 
resources,  facilities,  accommodations,  and  capability  to  effectively  setup  and  execute  the 
camp.  Some  of  the  camp  administration  activities,  such  as  coordinating  the  career  fair 
and  reserving  accommodations,  required  significant  preplanning,  time  and  effort.  Having 
an  application  would  help  to  ensure  the  host  is  aware  of  the  level  of  effort  as  well  as 
ensure  they  can  adequately  support  the  camp. 

9.  Group  size.  The  size  of  the  student  group  varied  across  each  of  the  camps.  However,  the 
feedback  from  most  of  the  interviewees  indicated  that  students  preferred  larger  camps 
(100+)  that  were  divided  into  smaller  team  groups  (20-25).  This  allowed  for  a  larger 
networking  opportunity  but  also  allowed  for  smaller  classes  and  more  teambuilding 
within  the  smaller  work  groups.  Groups  that  were  significantly  larger  noted  difficulties  in 
maintaining  the  attention  of  the  group  whereas  smaller  groups  noted  difficulty  managing 
the  students  as  different  levels  of  skill  and  pace.  Moreover,  many  of  the  interviewees 
noted  that  larger  camps  broken  into  smaller  working  groups  which  attended  the  various 
courses  each  day,  meaning  each  of  the  camp  courses  would  be  offered  to  a  different 
student  group  each  day  of  the  week,  would  provide  both  a  better  learning  environment 
and  a  better  teaching  environment. 

10.  Student  vetting.  The  MD  camp,  which  hosted  many  High  School  students,  strongly 
recommended  using  basic  filtering  for  identifying  students  for  camp  attendance.  Students 
for  the  MD  Camp  were  recruited  from  high  school  AP  CS  and  robotics  courses,  the 
CISCO  Academy,  and  Cyber  Foundations.  The  MD  interviewees  noted  that  managing  the 
selection  of  students  and  utilizing  a  structured  vetting  process  for  accepting  students 
greatly  increased  the  quality  and  caliber  of  those  in  attendance.  However,  the  MD 
interviewees  did  note  that  vetting  based  on  skills,  such  as  head  hunting  computer  science 
students,  didn’t  necessarily  make  better  students.  Simply,  identifying  students  who  were 
likely  to  engage  increased  the  overall  value  of  the  experience. 

11.  Teachers  as  students.  The  MD  camp  allowed  a  number  of  teachers  into  the  classes  to 
participate  as  students  and  found  that  to  be  a  great  success.  Allowing  teachers  to 
participate  alongside  students  helped  increase  student  interaction  and  engagement  and 
hopefully  maximized  the  value  of  the  content  teachers  could  take  back  to  their 
classrooms. 

12.  Camp  footprint.  Many  of  the  interviewees  noted  that  opening  the  enrollment  to  regional 
applicants  would  help  ensure  the  highest  quality  of  participant  possible. 

13.  Field  Trips.  Some  of  the  201 1  camps  hosted  field  trips  and  many  who  did  not  suggested 
the  idea  for  2012.  However,  field  trips  were  met  with  mixed  reviews.  On  the  positive  side 
it  gave  students  the  opportunity  to  visit  interesting  locations  and  real  world  security 
operation  centers.  On  the  negative  side,  if  the  location  didn’t  interest  all  the  students  it 
was  significant  overhead  for  an  activity  which  may  not  have  added  significant  value. 
Furthermore,  the  additional  of  the  field  trip  results  in  a  lost  day  of  education,  which  many 
interviewees  felt  was  more  important  for  the  camp. 

14.  Centralized  online  system.  Many  of  the  interviewees  noted  that  a  centralized  system  for 
registering  students,  coordinating  course  materials,  delivering  read  ahead  materials  and 
logistic  information,  configuring  CTF,  and  providing  a  virtual  community  space  for 
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students  to  collaborate  and  network  after  the  camp  concludes  would  be  of  tremendous 
benefit  to  all  the  stakeholders  of  the  camp.  Some  interviewees  also  expressed  value  in 
linking  course  instructor  profiles,  specific  course  materials  and  outlines,  as  well  as 
student  developed  work  products  into  the  online  system  as  well. 

15.  Press  &  Publicity.  Each  of  the  camps  had  varying  levels  of  success  in  obtaining 
publicity  for  their  camp.  Most  noted  difficulties  in  publicizing  the  event  with  local  media 
and  in  reaching  key  government  officials  to  attend.  However,  the  CA  camp  for  example 
was  successful  in  getting  CNN  television  coverage  as  well  as  articles  in  the  LA  Times. 

16.  Day  camp  versus  overnight.  The  2011  camps  were  a  mix  of  day  camps  and  overnight 
camps.  The  general  consensus  among  interviewees  was  that  the  day  camps  helped  reduce 
the  burden  of  logistics  and  helped  manage  camp  cost,  however  some  felt  that  having 
students  ‘in  residence’  increased  the  impact  of  the  material,  increased  the  level  of  team 
building,  and  maximized  the  students  time  at  the  camps. 

17.  Evening  hangout.  For  the  camps  which  were  overnight  camps  the  importance  of  a 
common  area  for  students  to  congregate  after  camp  hours  was  highly  important.  The 
interviewees  sighted  this  as  a  time  when  students  worked  together  on  course  issues, 
planned  for  team  activities  and  generally  networked  with  each  other.  In  nearly  all  cases 
where  this  space  was  not  available  pre-camp,  during  the  camp  there  was  a  push  to  make  it 
available. 

18.  Ethics  panel.  Nearly  every  interviewee  mentioned  the  success  of  the  ethics  panel  and  the 
student’s  satisfaction  with  that  particular  element  of  the  week.  However,  many 
interviewees  noted  the  ethics  panel  should  be  given  more  time,  particularly  because  of  the 
positive  student  reaction  to  the  activity.  Students  in  many  of  the  camps  enjoyed  speaking 
directly  with  law  enforcement  in  a  safe  environment,  which  provided  an  opportunity  to 
ask  questions  about  activities  which  fall  into  the  ‘gray  area’. 

19.  Surveys.  In  general,  the  feedback  on  the  survey  collection  was  positive.  However,  some 
interviewees  noted  that  it  was  particularly  important  to  ask  students  to  take  surveys  daily 
and  not  wait  until  the  end  of  the  week  to  have  them  all  completed.  Furthermore,  there 
were  some  concerns  about  the  repetitive  nature  of  some  of  the  data  in  the  surveys, 
particularly  for  students  that  completed  the  surveys  on  a  daily  basis. 

20.  Poster  session.  The  CA  camp  suggested  the  possibility  of  a  poster  session  or  other  venue 
for  students  to  highlight  their  work  and  activities. 

21.  Name  tags.  One  interviewee  who  attended  multiple  camps  noted  that  coordination 
activities  as  simple  as  providing  all  TAs,  instructions  and  students  with  name  tags 
improved  the  interaction  of  camp  participants  significantly. 

22.  CPE.  A  number  of  interviewees  mentioned  the  potential  value  in  certifying  the  camps  as 
Continuing  Professional  Education  credits  for  varying  certifications  in  the  field.  This 
would  add  another  layer  of  benefit  to  participants  of  the  camps. 

23.  Capture  The  Flag  (CTF).  All  the  interviewees  discussed  the  CTF  portion  of  the  event 
and  how  their  particular  setup  worked.  Feedback  regarding  CTF  has  been  included  in  the 
CTF  Data  Collection  Analysis  document. 

4.2  2011  Capture  the  Flag  (CTF)  Summary 

In  previous  IJSCC  summer  camps,  Capture  the  Flag  (CTF)  environments  were  built  with 
support  of  various  sponsors  which  while  successful  were  deemed  not  to  be  a  scalable  CTF 
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environment.  As  a  result,  USCC  explored  the  question  ‘which  virtual  environment  should  be 
used  for  the  “Capture  the  Flag  (CTF)”  for  the  camps?  And  what  should  be  the  partnership(s)  for 
the  future?”  Appendix  D,  “2011  CTF  Data  Collection  Analysis,”  outlines  the  technical 
environments  tested  in  the  camps  and  survey  results  from  the  summer  camp  hosts  regarding  the 
technical  aspects  of  each  solution. 

4.2.1  Common  Interview  Findings.  Across  all  the  interviews  a  number  of  common  themes 
were  captured  regarding  the  2011  Cyber  Camps  CTF  implementations.  The  list  below  explains 
each  of  these  common  elements.  Listed  numerically  for  ease  of  reference,  but  they  are  not  in  a 
particular  order. 

1 .  Infrastructure.  One  of  the  most  common  comments  from  interviewees  was  the  need  to 
ensure  the  hosting  facility  had  the  necessary  infrastructure  to  support  the  camp  and  the 
CTF.  This  includes  power  outlets  for  computers,  access  points  and  bandwidth  for  internet 
connections,  as  well  as  system  software,  operating  systems  and  patches  to  support  the 
tools  being  used.  Development  of  a  baseline  infrastructure  requirement  checklist  would 
be  beneficial  for  hosts  to  evaluate  their  space. 

2.  Host  and  TA  familiarity  the  CTF  tool,  scenario,  and  setup.  Most  interviewees  noted 
some  correlation  between  the  level  of  the  CTF  events  success  and  the  preparedness  of 
those  helping  to  host  the  event.  Those  who  had  representatives  from  the  tool’s  team  noted 
significant  benefit  in  executing  the  event. 

3.  CTF  preparation  time.  It  was  suggested  that  the  CTF  setup  team  needs  adequate  time  to 
setup  the  hardware/  environment,  install  necessary  software  as  well  as  test  key  elements 
of  the  configuration  with  enough  time  to  troubleshoot  if  any  problems  arise.  Conducting 
setup  the  morning  of  the  CTF  event  may  not  provide  enough  troubleshooting  time,  and  it 
was  suggested  the  CTF  be  held  in  a  different  space  from  the  camp  to  allow  the  team  to 
setup  the  day  before  the  CTF  event. 

4.  CTF  tool  documentation.  Many  of  the  interviewees  felt  there  was  a  lack  of  adequate,  in- 
depth  documentation  on  the  usage,  implementation,  execution,  and  configuration  of  the 
various  CTF  tools. 

5.  On-site  expertise.  A  number  of  the  interviewees  also  expressed  some  concern  about  the 
use  of  a  virtual  solution  without  an  on-site  subject  matter  expert  to  assist  in  facilitating 
and  troubleshooting  the  event  and  tool.  Having  that  on-site  resource  would  help  to  ensure 
any  technical  and  connectivity  issues  were  quickly  resolved,  but  would  also  help  in 
assuring  the  CTF  environment  and  scenario  are  correctly  setup  and  relayed  to 
participants. 

6.  CTF  debrief.  It  was  recommended  that  following  the  CTF  event,  participants  be  given 
the  opportunity  to  debrief  regarding  the  setup,  exploits,  processes  and  activities  that  were 
included  in  the  scenario.  This  would  help  participants  understand  what  elements  of  the 
CTF  scenario/  setup  they  may  have  missed  or  not  fully  leveraged.  Moreover,  providing  a 
time  for  campers  to  discuss  with  each  other  the  techniques  they  used  in  the  scenario 
would  assist  in  the  networking  aspect  of  the  camp. 

7.  Physical  environment  configuration.  It  was  noted  that  the  physical  setup  of  the  room 
for  hosting  the  CTF  may  have  had  an  impact  on  maximizing  the  benefit  of  the  event. 
Specifically,  those  who  hosted  in  the  CTF  in  an  auditorium  may  have  had  difficulty 
provided  participants  with  privacy  and  team  space  to  execute  the  competition.  Those  who 
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offered  smaller  tables  noted  that  CTF  teams  were  able  to  work  together  effectively  with 
some  privacy. 

8.  CTF  logistics.  Interviewees  discussed  a  number  of  logistical  challenges  with  the  CTF. 
From  physical  space  for  CTF  equipment  to  power  resources  to  bandwidth,  to  physical 
workspace  for  participants  to  sit  the  logistics  of  the  event  presented  most  camps  with 
some  form  of  complication. 

9.  CTF  prize.  It  was  mentioned  that  some  of  the  camp/  CTF  participants  were  not  students 
or  were  not  at  a  stage  in  their  academic  career  where  a  scholarship  was  the  most  effective 
prize  for  the  CTF  event.  It  was  suggested  that  in  addition  or  in  lieu  of  a  scholarship  other 
equally  valuable  awards,  such  as  professional  group  memberships,  be  considered. 

5.0  CONCLUSIONS 

The  success  of  this  effort  and  the  assessment  framework  is  dependent  on  the  continual 
flow  of  data  from  user  participation.  Although  there  is  likely  a  small  population  with  an  innate 
interest  in  continued  participation  over  time,  long-term  analysis  benefits  from  the  ‘pathway’ 
model  of  incremental  steps  to  include  activities,  education,  achievement,  and  ultimately  a 
benefit,  which  is  the  continual  achievement  and  success  yielding  a  payout  or  benefit  to  the 
participating  individual.  The  following  are  specific  conclusions  resulting  from  year’s  specifically 
involving  the  research  period.  Additionally,  the  USCC  has  conducted  the  high  school 
competitions  titled,  “Cyber  Foundations.”  The  results  of  the  first  year  competitions  were 
included  in  the  quarterly  status  reports  for  this  research  effort. 

5.1  Next  Steps  for  Cyber  Camps  2012 

From  the  pilot  camps  in  2010  to  the  2011  camps,  a  number  of  significant  improvements 
were  implemented  as  reflected  in  both  the  survey  data  and  interviews.  Additionally,  the  maturing 
of  the  camps  facilitates  better  and  more  extensive  feedback  to  continually  refine  the  camps. 

Based  on  the  analysis,  interviews  and  general  findings  there  are  a  number  of 
recommendations  for  considerations  during  the  2012  Cyber  Camps. 

Listed  numerically  for  ease  of  reference,  but  they  are  not  in  particular  order. 

1 .  Develop  a  camp  host  application  outlining  the  responsibilities  of  the  host,  the  facility  and 
logistic  requirements  as  well  as  the  expectations  for  host  participation. 

2.  Develop  a  comprehensive  set  of  instruction  manuals  and  sample  materials  for  hosts, 
instructors  and  TAs  to  ensure  all  the  roles  and  responsibilities  as  well  as  behavior 
exceptions  and  expected  outcomes. 

3.  Deploy  an  online  capability  to  facilitate  the  administration  of  camps,  registration  of 
participants,  coordination  of  activities,  delivery  of  materials  and  continuation  of 
relationships  through  a  virtual  community. 

4.  Leverage  the  online  capability,  develop  a  more  streamlined  feedback  and  survey 
mechanism  for  capturing  participant  experiences 

5.2  Next  Steps  for  the  CTF  for  Cyber  Camps  2012 

The  use  of  a  CTF  exercise  at  the  conclusion  of  the  Cyber  Camp  is  an  important 
component  of  each  camp  as  it  provides  participants  with  real  world,  hands  on  experience  in  the 
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field  of  cyber  security.  CTF  events  help  take  classroom  content  and  put  it  into  practice.  While 
the  intent  of  this  report  was  the  evaluation  of  CTF  technical  environments,  the  limited  findings 
resulted  in  some  common  CTF  issues  that  going  forward  USCC  will  consider  resolving  as  it  will 
increase  the  overall  CTF  experience  and  effectiveness. 

5.2.1  Common  CTF  Issues. 

1 .  Identifying  a  common  CTF  philosophy.  Across  the  CTF  exercises  there  are  two  general 
philosophies  that  have  been  taken.  The  first  philosophy  is  to  use  the  CTF  exercise  as  a 
concluding  assessment  or  exam  of  the  participant’s  absorption  of  the  week’s  material.  In 
this  approach  the  CTF  focused  specifically  on  the  activities,  techniques  and  material 
taught  throughout  the  week  and  served  as  a  test  of  the  week’s  materials.  The  second 
philosophy  is  to  develop  a  CTF  exercise  that  incorporated  the  elements  taught  throughout 
the  week,  but  also  provided  the  ability  for  students  go  further  and  apply  skills  and 
techniques  they  may  have  learned  elsewhere.  This  approach  pushed  students  beyond  the 
course  and  the  week’s  materials.  Identifying  a  consistent  USCC  approach  would  help 
better  define  CTF  objectives  and  help  standardize  the  event  across  camps. 

2.  Develop  a  CTF  infrastructure  requirements  checklist.  Each  of  the  various  setups  and 
tools  had  its  own  requirements  for  infrastructure  (power,  bandwidth,  physical  space, 
operating  system,  etc.).  Having  a  checklist  for  each  tool  would  allow  hosts  to  better 
assess  which  tools  would  best  operate  in  their  camps  physical  environment. 

3.  CTF  documentation  and  capability  assessment.  Each  of  the  various  tools  had  varying 
levels  of  documentation  and  preparation  support.  Working  with  vendors  to  ensure  host 
are  provided  with  enough  pre-event  instruction  and  with  enough  documentation  to  fully 
understanding  the  tool  would  ensure  the  CTF  event  runs  smoothly  and  effectively  and 
that  any  issues  that  arose  could  be  effectively  remediated. 

4.  Debrief  and  prizes.  Providing  a  post-CTF  debrief  to  explain  the  scenario,  the  applicable 
exploits,  techniques  that  could  have  been  used  and  areas  participants  overlooked  would 
help  provide  participants  with  a  better  understanding  of  the  event.  Furthermore, 
considering  alternative  prizes  for  CTF  winners  might  help  in  incentivizing  participants 
who  are  not  in  the  position  to  pursue  additional  education. 

In  summary,  there  are  two  key  data  points  to  highlight: 

•  For  student  vetting,  “Simply  identifying  students  who  were  likely  to  engage  increased  the 
overall  value  of  the  experience.”  This  statement  illustrates  the  use  of  competitions  such 
as  the  on-line  competitions  of  Cyber  Quests  for  the  1 8  years  and  older  participants  and 
the  Cyber  Foundations  for  the  high  school  participants  are  reaching  the  targeted 
population  and  addressing  the  short-term  objectives  of  identify,  engage  and  challenge. 

•  For  better  data  and  data  quality  for  the  assessment  framework,  “some  interviewees  noted 
that  it  was  particularly  important  to  ask  students  to  take  the  surveys  daily  and  not  wait 
until  the  end  of  the  week  to  have  them  all  completed.”  Additionally,  the  questions 
included  in  the  surveys  will  need  to  be  further  evaluated  to  ensure  they  are  a 
measurement  of  the  quality  of  the  experience. 

The  USCC  will  continue  to  use  the  social  media  capabilities  and  other  communications 
capabilities  as  well  the  resources  of  the  Center  for  Internet  Security  to  include  the  Multi-State 
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ISAC  community  to  increase  the  overall  numbers  of  participations  while  working  with  other 
competitions  to  broaden  the  information  and  data  for  the  assessment  framework. 
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The  table  below  provides  a  brief  overview  of  the  types  of  activities  taking  place  within  each  time  horizon 
at  each  assessment  period. 


Short-Term 

Mid-Term 

Long-Term 

Pre-Event 

In  the  short-term,  pre-event 
data  collection  will  only 
occur  using  previously 
established  registration 
processes  for  each  event 

In  the  mid-term,  the 
assessment  framework 
calls  for  working  with 
events  to  incorporate  the 
baseline  data  elements  for 
each  participant  into  the 
existing  registration 
process. 

Long  term,  the  USCC 
participant  platform  will 
provide  a  consolidated 
platform  for  event 
registration,  user  data 
collection  and  a  roaming 
profile  of  user  information 
and  activity. 

In-Event 

In  the  short-term,  in-event 
data  collection  will  only 
occur  using  previously 
established  processes  for 
each  event,  if  such  a 
collection  mechanism 
already  exists  for  that 
event. 

In  the  mid-term  in-event 
data  collection  will  only 
occur  using  previously 
established  processes  for 
each  event.  This  may  be 
supplemented  by 
capabilities  incorporated 
into  the  basic  toolkit 
provided  by  USCC. 

Long-term,  a  collection  of 
tools  and  technologies  will 
be  provided  by  the  USCC 
participant  platform  to  aid 
in  the  real  time  collection 
of  data  during  events. 

These  tools  will  assist 
events  in  collecting  data 
and  participant  feedback  as 
it  happens  to  hone  in  on 
specific  components  of 
events  that  need 
improvement. 

Post-Event 

In  the  short-term,  the 
assessment  framework 
calls  for  working  with 
specific  events  to 
coordinate  and  baseline  the 
questions  and  data  being 
collected  about  participant 
satisfaction  and  future 
plans  with  respect  to  the 
event.  This  will  be 
supplemented  by  the  use  of 
an  existing  third-party  data 
collection  and  analysis 
platform. 

In  the  mid-term,  post-event 
data  collection  will  include 
tools  and  technologies 
provided  by  USCC  tool 
kits  for  the  collection, 
compilation  and  analysis 
of  participant  feedback. 

This  will  allow  for  basic 
event  outcome  analysis 
and  provide  feedback  to 
event  coordinators.  In  the 
mid-term  this  may  also  be 
supplemented  by  the  use  of 
an  existing  third-party  data 
collection  and  analysis 
platform. 

Long-term,  the  USCC 
participant  platform  will 
provide  the  capability  to 
automate  post-event  follow 
up  at  varying  intervals  (3, 

6,  9  months  after)  to 
provide  persistent 
monitoring  of  participants. 
Further,  this  platform  will 
assist  in  conducting 
analysis  of  a  variety  of 
events  across  USCC 
purview. 

The  following  sections  layout  some  of  the  specific  objectives,  questions,  data,  methodologies, 
technologies  utilized  within  each  time  horizon. 
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Short-Term  Assessment  Methodology 

The  objectives  of  the  assessment  framework  in  the  short-term  are  geared  toward  identifying  the  key  data 
elements,  deriving  assessment  baselines  and  establishing  the  processes  by  which  data  will  be  collected 
over  the  mid-  and  long-terms.  The  table  below  provides  an  overview  of  the  key  objectives,  data  and 
collection  methods  for  each  of  the  overarching  focus  areas  in  the  short-term. 


Pre-Event 

In-Event 

Post-Event 

Assessment 

Objectives 

(Metrics) 

•  %  of  new  users  attending 
USCC  events  (as  a  percent 
of  total  participants) 

•  %  of  returning  users 
attending  USCC  events  (as  a 
percent  of  total  participants) 

•  #  of  High  Schools 
participating  in  USCC 
activities  (including 
institution  information) 

•  #  of  Universities 
participating  in  USCC 
activities  (including 
institution  information) 

•  #  of  women  attending  USCC 
events 

•  #  of  minorities  attending 
USCC  events 

•  Geographical  information  on 
participants 

• 

•  %  of  monthly  unique  visitors 
to  the  USCC  website  (unique 
hits  versus  total  hits) 

•  %  of  monthly  repeat  visitors 
to  the  USCC  website  (repeat 
hits  as  percent  of  total) 

•  %  of  users  downloading  free 
web  tutorials 

•  %  of  users  using  toolkits 

•  #  of  users  participating  in 
USCC  web  games 

•  #  of  users  participating  in 
‘in-person’  competition 

Questions  to 
be  Answered 

•  Are  individuals  engaging 
USCC  and  related  cyber 
security  events? 

•  What  types  of  individuals 
are  engaging  these  events? 

•  What  forms  of  in-event 
data  collection  are  non- 
intrusive  yet  effective? 

•  What  capabilities/  tools  are 
participants  using? 

•  What  are  the  capabilities  and 
requirements  for  inclusion  as 
a  USCC  recognized  system, 
activity,  initiative? 

Data 

Elements 

•  Number  of  Registrants 

•  Number  of  Participants 

•  Participant  Gender 

•  Participant  Ethnicity 

•  Participant  Age 

•  Participant  GPA/  Education 

•  Participant  Career  Interests 

•  Participant  Proficiencies 

•  Participant  Extracurricular 
activities 

O  Clubs,  Sports,  etc 

• 

•  Tool,  feature  and  usage 
analysis 
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Collection 

Methods 

•  Demographics  surveys 

•  Rudy’s  Survey 

•  Galen’s  Test  Group 

• 

Success 

Metrics 

•  Development  of  baseline 
demographic  participation 
metrics 

• 

•  (immediate  post-event 
assessments) 

Technologies 

(Toolkit) 

•  Registration  forms 

•  Participant  profiles 

O  Ex:  Facebook,  Twitter 

• 

•  SurveyMonkey 

In  the  short  term,  some  of  the  pre-event  data  may  only  be  collectable  in  post-event  surveys. 
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Mid-Term  Assessment  Methodology 

The  objectives  of  the  assessment  framework  in  the  mid-term  are  geared  toward  developing  the  feedback 
loop  to  implement  and  assess  improvements  to  the  process.  Moreover,  the  mid-term  is  the  transition  point 
to  develop  a  more  robust  infrastructure  for  data  collection,  user  management  and  the  development  of  a 
cyber  security  social  network.  The  mid-term  effort  is  collectively  a  transition  and  honing  period  to 
establish  the  in-house  infrastructure  to  support  this  effort’s  long  term  objectives  as  well  as  to  refine  the 
baseline,  questions  and  assessment  mechanisms.  The  table  below  provides  an  overview  of  the  key 
objectives,  data  and  collection  methods  for  each  of  the  areas  in  the  mid-term. 


Pre-Event 

In-Event 

Post-Event 

Assessment 

Objectives 

(Metrics) 

•  Participant  goals 

•  Participant  evaluation  of 
the  event  quality 

•  Likert-scale  ratings  of  task 
absorption 

•  Likert-scale  ratings  of 
perceived  task  utility 

•  Participant  engagement 
assessment 

• 

Questions  to 
be  Answered 

•  What  are  participant 
expectations  for  the  event? 

•  What  are  participant 
expectations  regarding  the 
rigor,  reputation  and  value 
of  the  event? 

•  How  engaged  are 
participants  during  the 
event 

• 

•  Did  the  event  meet  or 
exceed  participant 
expectations? 

•  Do  participants  utilize  and 
continue  to  leverage  USCC 
provided  toolkits,  systems, 
infrastructure,  etc? 

Data 

Elements 

• 

• 

• 

Collection 

Methods 

• 

•  Rudy’s  Survey 

•  Galen’s  Test  Group 

•  USCC  Participant  Platform 

•  Automated  periodic 
surveys 

•  Automated  periodic 
knowledge  assessments 
(test  or  games) 

Success 

Metrics 

•  Participant  pre-event 
perception  of  events 
(marketing  and  outreach 
feedback) 

• 

•  Baseline  of  post-event 
usage  and  retention  (~3 
month  post-event 
assessment) 

Technologies 

(Toolkit) 

• 

• 

• 
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Long-Term  Assessment  Methodology 

The  objectives  of  the  assessment  framework  in  the  long-term  are  geared  toward  the  deployment  of 
sponsored  collection  platforms,  continual  refinement  of  data  collection  and  analysis,  participant  tracking, 


Pre-Event 

In-Event 

Post-Event 

Assessment 

Objectives 

(Metrics) 

• 

•  Participant  in-event 
material  comprehension 

•  Participant  in-event,  real¬ 
time  feedback 

•  %  of  participants  engaged 
in  future  activities 

Questions  to 
be  Answered 

•  What  is  the  participant’s 
assessment  of  the  events 
reputation  and  quality? 

•  Do  participants  understand 
the  material? 

•  Is  the  level  of  the  material 
challenging  and 
appropriate? 

•  Are  participants  retaining 
knowledge  and  skills 
acquired  at  each  event? 

•  Do  participants  engage  in 
other  USCC  or  similar 
cyber  security  events? 

•  Do  participants  pursue 
careers  in  cyber  security 
fields? 

Data 

Elements 

•  Participant’s  history  with 
USCC  and  similar  cyber 
security  events 

•  Participant’s  maximum  time 
investment 

• 

• 

Collection 

Methods 

• 

• 

•  Automated  periodic 
surveys 

•  Automated  periodic 
knowledge  assessments 
(test  or  games) 

•  Periodic  participant 
interviews 

Success 

Metrics 

•  Understanding  of  participant 
expectations  based  on 
background  and  experience 

•  Baseline  for  assessing  event 
marketing  and  research 

• 

•  Baseline  of  post-event 
usage  and  retention  (3,6, 

12  month  post-event 
assessments) 

Technologies 

(Toolkit) 

• 

• 

• 
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Short-Term 

In  the  short-term  phase  of  the  of  USCC  assessment  methodology,  USCC  will  utilize  an  online  survey  and 
data  management  platform,  SurveyMonkey  (www.surveymonkey.com),  to  conduct  post-event  surveys 
and  data  collection.  Using  this  platform  USCC  will  be  able  to  inform  the  objectives  and  collect  the  data 
elements  called  out  in  the  Assessment  Framework  Methodology  including  those  in  Appendix  A. 

2011  Participant  Survey  Question  Outline 

1 .  Participant  Information 

1.1.  What  is  your  first  and  last  name?  [* required,  free  form  text  field] 

1.2.  What  is  your  email  address?  [* required ,  free  form  text  field  w/  email  format  validation] 

2.  Instructor  &  Event  Evaluation 

2.1.  What  USCC  event  did  you  attend?  [* required ,  drop  down  menu  w/  other  option] 

2. 1.1.  Course/  event  list  specific  to  each  USCC  event 

2.2.  Is  this  your  first  USCC  event?  [yes/  no  selection] 

2.3.  Where  did  you  first  hear  about  this  event?  [drop  down  menu  w/  other  option] 

2.3. 1  .Email/  Newsletter 

2.3.2. Facebook 

2. 3. 3.  Family  or  Friend 

2. 3. 4.  Magazine  Article 

2. 3. 5.  Newspaper  Story 

2. 3. 6.  Professor 

2.3.7. TV  News 

2. 3. 8.  Twitter 

2. 3. 9.  University  Publication 

2.3.10.  Website/  Search  Engine 

2.3.11.  YouTube 

2.4.  On  a  scale  of  1-10,  what  is  your  overall  evaluation  of  this  course/  event?  [drop  down  menu] 

2.4.1. 1,  Bad 

2.4.2. 2, 

2.4. 3. 3,  Poor 

2.4.4. 4, 

2.4.5. 5,  Marginal 

2.4.6. 6, 

2.4.7. 7,  Good 

2.4.8. 8, 

2.4.9. 9,  Great 
2.4.10.  10,  Excellent 

2.5.  On  a  scale  of  1-10,  what  is  your  overall  evaluation  of  the  instructor's  teaching  skill?  [drop  down 
menu] 

2.5. 1.1,  Bad 

2. 5.2. 2, 

2. 5. 3. 3,  Poor 

2. 5.4. 4, 

2. 5. 5. 5,  Marginal 

2.5.6. 6, 
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2. 5. 7. 7,  Good 

2.5. 8.8, 

2.5.9. 9,  Great 

2.5.10.  10,  Excellent 

2.6.  On  a  scale  of  1-10,  what  is  your  overall  evaluation  of  the  value  of  the  course  content?  [drop 
down  menu] 

2.6.1. 1,  Bad 

2. 6.2. 2, 

2. 6. 3. 3,  Poor 

2. 6. 4. 4, 

2.6.5. 5,  Marginal 

2.6.6. 6, 

2.6.7. 7,  Good 

2.6.8. 8, 

2.6.9. 9,  Great 

2.6.10.  10,  Excellent 

2.7.  What  encouraged  you  to  participate  in  this  event?  [free  form  memo  field] 

3.  Participant  Demographics 

3.1.  What  is  your  age?  [free  form  text  field  w/  numeric  format  validation] 

3.2.  What  is  your  gender?  [male/  female  selection] 

3.3.  What  is  your  primary  city  of  residence?  [free  form  text  field] 

3.4.  What  is  your  primary  state  of  residence?  [drop  down  menu  of  US  States] 

3.5.  Which  best  describes  your  ethnicity?  [drop  down  menu] 

3.5.1  .American  Indian  or  Alaska  Native 

3. 5.2.  Asian 

3. 5. 3.  Black  or  African  American 

3. 5. 4.  Hispanic  or  Latino 

3.5.5. Native  Hawaiian  or  Other  Pacific  Islander 

3. 5. 6.  White 

4.  Education  Information 

4.1.  Which  option  best  describes  your  current  level  of  education?  [drop  down  menu] 

4. 1 .  E Currently  in  pre-High  School 

4. 1.2.  Currently  in  High  School 

4. 1.3.  Completed  High  School 

4. 1.4.  Currently  in  Undergrad  Program 

4. 1 .5.  Completed  Undergrad  Program 

4. 1.6.  Currently  in  Graduate  Program 

4. 1.7.  Completed  Graduate  Program 

4. 1.8.  Currently  in  Doctoral  Program 

4. 1.9.  Completed  Doctoral  Program 

4.2.  What  is  the  name  of  your  current  academic  institution?  (high  school,  university,  college,  etc) 
[free  form  text  field] 

4.3.  What  is  your  current  grade  point  average  (on  a  4.0  scale)?  (if  unknown,  please  approximate) 
[free  form  text  field  w/  numeric  format  validation] 

4.4.  What  is  your  current  major  or  primary  area  of  study?  (if  applicable)  [free  form  text  field] 

4.5.  What  do  you  consider  to  be  your  two  greatest  academic  strength  areas?  (e.g.,  math,  science, 
computers,  English,  art,  etc)  [free  form  memo  field] 
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4.6.  What  types  of  extracurricular  activities,  if  any,  are  you  currently  engaged  in?  (e.g.,  soccer,  chess 
club,  student  government,  computer  club,  etc)  [free  form  memo  field] 

5.  Cyber  Security  Proficiency 

5.1.  Have  you  ever  had  formal  training/  coursework  on  cyber  security  topics?  [yes/  no  selection] 

5.2.  On  a  scale  of  1-10,  what  is  your  level  of  familiarity/  comfort  with  the  field  of  cyber  security? 
[drop  down  menu] 

5.2.1. 1,  No  Familiarity 
5.2.2. 2 

5. 2. 3. 3,  Novice 
5. 2. 4. 4 

5.2.5. 5,  Basic  Familiarity 
5.2.6. 6 

5.2.7. 7,  Proficient 
5.2.8. 8 

5.2.9. 9,  Highly  Proficient 
5.2.10.  10,  Expert 

5.3.  If  you  have  a  background  in  cyber  security,  what  areas  of  cyber  security  do  you  consider 
yourself  most  proficient  in?  (e.g.,  system  administration,  programming,  forensics,  system 
penetration,  etc.)  [free  form  memo  field] 

6.  Competition/  Event  History 

6.1.  Where  did  you  first  hear  about  the  USCC  organization?  [drop  down  menu  w/  other  option] 

6.1. 1  .Email/  Newsletter 

6.1.2. Facebook 

6. 1.3.  Family  or  Friend 

6. 1.4.  Magazine  Article 

6. 1.5.  Newspaper  Story 

6. 1.6.  Professor 

6. 1.7.  TV  News 

6. 1.8.  Twitter 

6. 1 .9.  University  Publication 

6.1.10.  Website/  Search  Engine 

6.1.11.  YouTube 

6.2.  Have  you  engaged  in  cyber/  security  activities,  events,  or  competitions  in  the  past?  [yes/  no 
selection] 

6.3.  If  you  have  engaged  in  similar  events  in  the  past,  briefly  list  these  previous  events  and  activities, 
[free  form  memo  field] 

7.  Professional  Interests 

7.1.  What  is  your  desired/  planned  career  field?  [free  form  text  field] 

7.2.  Have  you  previously  considered  a  career  in  cyber  security?  [yes/  no  selection] 

7.3.  On  a  scale  of  1-5,  after  your  participation  in  this  event  is  what  is  the  likelihood  you  will  to 
explore  cyber  security  as  a  career  path?  [drop  down  menu] 

7.3. 1.1,  Not  At  All 

7. 3.2. 2,  Unlikely 

7.3.3. 3,  Likely 

7. 3. 4. 4,  Very  Likely 

7. 3. 5. 5,  Certainly 
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8.  Closing 

8.1.  If  you  have  any  additional  comments  or  feedback,  please  provide  it  here,  [free  form  memo  field] 

2011  CTF  Survey  Question  Outline 

1 .  Event  Setup 

1.1.  Was  the  vendor  helpful  in  explaining  the  set  up  for  this  event?  [yes/  no  selection  w/  detail  memo 
field  option] 

1.2.  On  a  scale  of  1-5,  how  would  you  rate  the  quality  of  the  technical  documentation  provided  by  the 
vendor?  [drop  down  menu  w/  detail  memo  field  option] 

1.2.1. 1,  Inadequate 

1.2.2. 2,  Poor 

1.2. 3. 3,  Acceptable 

1. 2.4.4,  Good 

1.2. 5. 5,  Excellent 

1 .3.  Did  the  competition  appear  to  provide  the  technical  challenge  for  the  class?  [yes/  no  selection  w/ 
detail  memo  field  option] 

1 .4.  Do  you  feel  the  technical  set  up  was  easy  to  achieve?  [yes/  no  selection] 

1.5.  Did  the  technical  performance  meet  your  expectations?  [yes/  no  selection] 

2.  Evaluation 

2.1.  On  a  scale  of  1-10,  what  is  your  overall  evaluation  of  this  course/  event?  [drop  down  menu] 

2.1. 1.1,  Bad 

2. 1.2. 2, 

2. 1.3. 3,  Poor 

2. 1.4. 4, 

2. 1.5.5,  Marginal 

2. 1.6. 6, 

2. 1.7.7,  Good 

2. 1.8.8, 

2. 1.9. 9,  Great 

2.1.10.  10,  Excellent 

2.2.  On  a  scale  of  1-10,  what  is  your  evaluation  of  the  scoreboard  display?  [drop  down  menu] 

2.2. 1.1,  Bad 

2.22.2, 

2.2. 3. 3,  Poor 

2.2.4. 4, 

2.2. 5. 5,  Marginal 

2.2. 6. 6, 

2.2.7. 7,  Good 

2.2.8. 8, 

2.2.9. 9,  Great 

2.2.10.  10,  Excellent 

2.3.  On  a  scale  of  1-10,  what  is  your  evaluation  of  the  network  display  for  the  TAs?  [drop  down 
menu] 

2.3. 1.1,  Bad 

2.3.2. 2, 

2. 3. 3. 3,  Poor 

2. 3.4.4, 
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2. 3. 5. 5,  Marginal 

2. 3. 6. 6, 

23.1.1 ,  Good 

2.3. 8.8, 

2. 3. 9. 9,  Great 

2.3.10.  10,  Excellent 

2.4.  On  a  scale  of  1-10,  what  is  your  evaluation  of  the  ease  of  use  for  the  technical  solution?  [drop 
down  menu] 

2.4. 1.1,  Bad 

2.4.2. 2, 

2.4. 3. 3,  Poor 

2.4.4. 4, 

2.4.5. 5,  Marginal 

2.4.6. 6, 

2.4.7. 7,  Good 

2.4.8. 8, 

2.4.9. 9,  Great 

2.4.10.  10,  Excellent 

3 .  Event  Assessment 

3.1.  Would  you  use  this  solution  for  your  upcoming  activities?  [free  form  memo  field] 

3.2.  What  are  the  strengths  of  this  competition?  [free  form  memo  field] 

3.3.  What  are  the  areas  for  improvements  for  this  event/  competition?  [free  form  memo  field] 

3.4.  Would  you  recommend  this  event/  competition  to  your  peers?  [yes/  no  selection] 

4.  Closing 

4.1.  If  you  have  any  additional  comments  or  feedback,  please  provide  it  here,  [free  form  memo  field] 
Survey  Data  Structure 

Data  collected  via  the  SurveyMonkey  tool,  for  both  the  participant  survey  and  the  CTF  survey,  is 
maintained  in  raw  data  format  as  provided  by  SurveyMonkey.  The  data  headers  for  these  data  sets  are 
explained  below. 

Note;  Raw  data  is  provided  by  SurveyMonkey  in  both  .csv  and  .xls  formats  for  easy  use  and  ingest  into 
database  and  statistics  tools.  SurveyMonkey  also  provides  a  data  summary  which  provides  a  quick 
graphical  view  of  the  collected  data.  This  summary  report  has  also  included  in  both  .html  and  .pdf 
formats. 

2011  Participant  Survey  Data  Structure 

1 .  RespondentID ,  numeric,  this  is  a  unique  identifier  for  each  survey  completion  automatically 
generated  by  SurveyMonkey  for  each  completed  survey. 

2.  CollectorlD ,  numeric,  collectors  are  a  feature  of  SurveyMonkey  which  facilitate  the  completion 
of  the  same  survey  though  different  entry  points.  This  feature  is  not  applicable  to  the  2011  camp 
data  and  each  camp  data  set  should  have  the  same  CollectorlD. 

3.  StartDate ,  date  time,  represents  the  date  the  participant  began  taking  the  survey. 

4.  EndDate ,  date  time,  represents  the  date  the  participant  completed  taking  the  survey. 

5.  IP  Address ,  text,  represents  the  computer  address  from  which  a  participant  completed  the  survey. 

6.  Email  Address ,  text,  this  is  a  standard  SurveyMonkey  field  which  is  not  used  in  this  survey. 
Participant  email  information  is  collected  in  1 1  below. 
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7.  First  Name ,  text,  this  is  a  standard  SurveyMonkey  field  which  is  not  used  in  this  survey.  First 
name  information  is  collected  in  10  below. 

8.  LastName ,  text,  this  is  a  standard  SurveyMonkey  field  which  is  not  used  in  this  survey.  Last  name 
information  is  collected  in  10  below. 

9.  Custom  Data ,  text, 

10.  What  is  your  first  and  last  name?,  text,  is  the  provided,  combined  first  and  last  name  of  the 
participant  who  completed  the  survey. 

1 1 .  What  is  your  email  address ?,  text,  is  the  provided  email  address  of  the  participant  who  completed 
the  survey. 

The  remainder  of  the  data  headings  in  this  data  set  map  directly  to  the  questions  outlined  in  2011 
Participant  Survey  Question  Outline  above.  All  fields  except  GPA  and  age,  both  numerics,  are  collected 
as  text. 

2011  CTF  Survey  Data  Structure 

12.  RespondentID ,  numeric,  this  is  a  unique  identifier  for  each  survey  completion  automatically 
generated  by  SurveyMonkey  for  each  completed  survey. 

13.  CollectorlD ,  numeric,  collectors  are  a  feature  of  SurveyMonkey  which  facilitate  the  completion 
of  the  same  survey  though  different  entry  points.  This  feature  is  not  applicable  to  the  2011  camp 
data  and  each  camp  data  set  should  have  the  same  CollectorlD. 

14.  StartDate ,  date  time,  represents  the  date  the  participant  began  taking  the  survey. 

15.  EndDate ,  date  time,  represents  the  date  the  participant  completed  taking  the  survey. 

16.  IP  Address,  text,  represents  the  computer  address  from  which  a  participant  completed  the  survey. 

17.  Email  Address,  text,  this  is  a  standard  SurveyMonkey  field  which  is  not  used  in  this  survey. 
Participant  email  information  is  collected  in 

18.  First  Name,  text, 

19.  LastName,  text, 

20.  Custom  Data,  text, 

The  remainder  of  the  data  headings  in  this  data  set  map  directly  to  the  questions  outlined  in  2011  CTF 
Survey  Question  Outline  above.  All  fields  are  collected  as  text. 
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2011  Sample  Analysis 

Utilizing  the  short-term  collection  methodology  the  summer  2011  camps,  USCC  was  able  to  collect  360 
survey  results  records.  The  data  in  these  records,  as  outlined  above,  allows  USCC  to  conduct  analysis  on 
camp  participant  demographics,  background,  professional  interests  and  camp  experiences.  This  analysis 
will  conclude  with  the  2011  Competition  Data  Collection  Analysis.  Below  is  a  brief  sample  of  the  data 
collected  and  the  actual  analysis  completed  as  part  of  the  summary  report. 

Summary:  Total  Survey  Completions _ 
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Question  10:  What  is  your  age? 
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ICA 

IMD 

(MO 

IVA 

IDE 


CA 

MD 

MO 

VA 

DE 

Total 

Percent 

Less  than  15 

■ 

0 

■■ 

1 

■■ 

1 

0 

■■ 

0 

2 

■■ 

0.56% 

15 

0 

4 

0 

0 

0 

4 

1.11% 

16 

■ 

0 

7 

■■ 

0 

0 

0 

7 

■■ 

1.94% 

17 

0 

5 

0 

0 

0 

5 

1.39% 

18 

8 

3 

6 

■ 

0 

■■ 

1 

18 

5.00% 

19 

9 

0 

0 

1 

1 

11 

3.06% 

20 

5 

0 

■■ 

4 

2 

0 

11 

■■ 

3.06% 

21 

6 

0 

8 

2 

14 

30 

8.33% 

22 

9 

0 

■■ 

16 

4 

■■ 

2 

31 

■■ 

8.61% 

23 

0 

0 

11 

1 

10 

22 

6.11% 

24 

7 

0 

■■ 

0 

1 

0 

8 

■■ 

2.22% 

25 

4 

1 

11 

1 

7 

24 

6.67% 

26 

11 

0 

■■ 

4 

1 

7 

23 

6.39% 

27 

9 

0 

0 

2 

0 

11 

3.06% 

28 

3 

0 

■■ 

14 

1 

5 

23 

6.39% 

29 

5 

0 

0 

1 

1 

7 

1.94% 

30 

4 

0 

■■ 

7 

0 

13 

24 

■■ 

6.67% 

31 

4 

0 

2 

0 

2 

8 

2.22% 

32 

3 

■■ 

0 

■■ 

0 

0 

2 

5 

■■ 

1.39% 

33 

3 

0 

0 

1 

1 

5 

1.39% 

34 

0 

0 

■■ 

8 

1 

1 

10 

■■ 

2.78% 

35 

0 

0 

0 

0 

0 

0 

0.00% 

36 

0 

0 

■■ 

3 

1 

0 

4 

■■ 

1.11% 

37 

0 

0 

2 

1 

0 

3 

0.83% 

38 

0 

■■ 

0 

■■ 

4 

1 

0 

5 

■■ 

1.39% 

39 

3 

0 

0 

0 

0 

3 

0.83% 

40 

0 

0 

■■ 

0 

0 

0 

0 

0.00% 

Greater  than  40 

11 

1 

12 

2 

2 

28 

7.78% 

No  Response 

4 

1 

■■ 

8 

1 

14 

28 

■■ 

7.78% 

APPROVED  FOR  PUBLIC  RELEASE;  DISTRIBUTION  UNLIMITED. 

34 


uscc 

US  CYBER  CHALLENGE 


USCC  Assessment  Framework  Methodology 
Appendix  B:  Assessment  Database  Structure 


Question  15:  Which  option  best  describes  your  current  level  of  education? 
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Long-Term  Database  Structure 


Transitioning  through  the  mid-term  and  into  the  long-term  USCC  will  develop  a  more  sophisticated 
participant  management  platform  to  maintain  participant  profiles,  event  registration  data,  competition 
performance  metric,  and  feedback  data  among  other  key  elements.  The  USCC  participant  portal 
objectives  include; 

1 .  Establishment  of  a  virtual  community  workspace  for  USCC  participants  to  encourage  continued 
engagement  and  allow  participants  to  develop  a  virtual  portfolio  of  work. 

2.  Development  of  a  ‘roaming’  profile  to  allow  participants  to  track  their  involvement  and 
engagements  in  the  USCC  community.  Utilizing  the  profile  information,  the  platform  will  be 
able  to  recommend  events  to  attend  or  skill  areas  participants  should  work  to  develop  as  well  as 
match  participants  with  potential  mentors. 

3.  Establishment  of  a  consistent  participant  management  toolkit  for  hosting  events  and  supporting 
user  registration.  Organizers  can  create  pages  for  their  competitions  and  administrate  their  events 
through  the  participant  portal,  allowing  seamless  interaction  with  participant  profiles  and  portal 
information. 

4.  Consolidation  of  USCC  materials  such  as  event  descriptions,  course  outlines,  educator  profiles, 
event  pages,  recognition  systems  (badges,  titles,  achievements,  etc),  job  postings,  skill 
development  activities,  and  other  USCC  information. 

5.  Establishment  of  an  integrated  feedback  mechanism  which  will  collect  participant  data 
throughout  the  time  horizons  of  each  event.  This  component  will  help  track  participants  into  the 
professional  world  and  provide  critical  feedback  data  to  USCC. 

6.  Integration  with  external  systems  such  as  Facebook,  Twitter  and  competition  specific  systems  to 
allow  participants  to  easily  access  and  update  their  profiles. 

These  goals  will  be  achieved  by  developing  a  system  which  includes  the  general  elements  outlined  in  the 
diagram  below. 
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Underlying  the  participant  management  platform  will  be  a  database  which  encompasses  the  data  elements 
from  the  short-term  as  well  as  accommodates  data  from  the  capabilities  added  to  the  system  during  the 
mid  and  long-term  phases.  Below  is  a  representative  diagram  illustrating  the  core  data  structure  which 
will  contain  the  short-term  data  and  provide  the  basis  for  the  participant  management  platform’s  database 
into  the  mid  and  long-terms  phases. 
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Survey  Structures 


2011  Survey  Structures 

In  201 1  two  surveys  were  used  to  collect  the  necessary  feedback  from  participants  and  hosts.  The  first, 
2011  Participant  Survey ,  was  given  to  individuals  who  participated  in  the  event.  The  second,  2011  CTF 
Survey ,  was  given  to  administrators  who  participated  in  the  administration  of  the  Capture  the  Flag 
component  of  each  camp. 

2011  Participant  Survey  Structure 

9.  Participant  Information 

9.1.  What  is  your  first  and  last  name?  [* required '  free  form  text  field] 

9.2.  What  is  your  email  address?  [*required,  free  form  text  field  w/  email  format  validation] 

10.  Instructor  &  Event  Evaluation 

10. 1 .  What  USCC  event  did  you  attend?  [* required,  drop  down  menu  w/  other  option] 

10.1.1.  Course/  event  list  specific  to  each  USCC  event 

10.2.  Is  this  your  first  USCC  event?  [yes/  no  selection] 

10.3.  Where  did  you  first  hear  about  this  event?  [drop  down  menu  w/  other  option] 

10.3. 1 .  Email/  Newsletter 

10.3.2.  Facebook 

10.3.3.  Family  or  Friend 

10.3.4.  Magazine  Article 

10.3.5.  Newspaper  Story 

10.3.6.  Professor 

10.3.7.  TV  News 

10.3.8.  Twitter 

10.3.9.  University  Publication 

10.3. 10.  Website/  Search  Engine 

10.3.11.  YouTube 

10.4.  On  a  scale  of  1-10,  what  is  your  overall  evaluation  of  this  course/  event?  [drop  down  menu] 


10.4.1. 

1,  Bad 

10.4.2. 

2, 

10.4.3. 

3,  Poor 

10.4.4. 

4, 

10.4.5. 

5,  Marginal 

10.4.6. 

6, 

10.4.7. 

7,  Good 

10.4.8. 

8, 

10.4.9. 

9,  Great 

10.4.10 

.10,  Excellent 

10.5.  On  a  scale  of  1-10,  what  is  your  overall  evaluation  of  the  instructor's  teaching  skill?  [drop 
down  menu] 

10.5.1.  1,  Bad 

10.5.2.  2, 

10.5.3.  3,  Poor 

10.5.4.  4, 
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10.5.5. 

5, 

Marginal 

10.5.6. 

6, 

10.5.7. 

7, 

Good 

10.5.8. 

8, 

10.5.9. 

9, 

Great 

10.5.10 

.10,  Excellent 

10.6.  On  a  scale  of  1-10,  what  is  your  overall  evaluation  of  the  value  of  the  course  content?  [drop 
down  menu] 


10.6.1. 

1, 

Bad 

10.6.2. 

2, 

10.6.3. 

3, 

Poor 

10.6.4. 

4, 

10.6.5. 

5, 

Marginal 

10.6.6. 

6, 

10.6.7. 

7, 

Good 

10.6.8. 

8, 

10.6.9. 

9, 

Great 

10.6.10 

.1C 

i,  Excellent 

10.7.  What  encouraged  you  to  participate  in  this  event?  [free  form  memo  field] 

11.  Participant  Demographics 

11.1.  What  is  your  age?  [free  form  text  field  w/  numeric  format  validation] 

1 1 .2.  What  is  your  gender?  [male/  female  selection] 

11.3.  What  is  your  primary  city  of  residence?  [free  form  text  field] 

1 1 .4.  What  is  your  primary  state  of  residence?  [drop  down  menu  of  US  States] 

11.5.  Which  best  describes  your  ethnicity?  [drop  down  menu] 

11.5.1.  American  Indian  or  Alaska  Native 

11.5.2.  Asian 

11.5.3.  Black  or  African  American 

1 1 .5.4.  Hispanic  or  Latino 

11.5.5.  Native  Hawaiian  or  Other  Pacific  Islander 

11.5.6.  White 

12.  Education  Information 

12.1.  Which  option  best  describes  your  current  level  of  education?  [drop  down  menu] 

12.1.1.  Currently  in  pre-High  School 

12.1 .2.  Currently  in  High  School 

12.1.3.  Completed  High  School 

12.1 .4.  Currently  in  Undergrad  Program 

12.1.5.  Completed  Undergrad  Program 

12.1.6.  Currently  in  Graduate  Program 

12.1.7.  Completed  Graduate  Program 

12.1.8.  Currently  in  Doctoral  Program 

12.1.9.  Completed  Doctoral  Program 

12.2.  What  is  the  name  of  your  current  academic  institution?  (high  school,  university,  college,  etc.) 
[free  form  text  field] 

12.3.  What  is  your  current  grade  point  average  (on  a  4.0  scale)?  (if  unknown,  please  approximate) 
[free  form  text  field  w/  numeric  format  validation] 
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12.4.  What  is  your  current  major  or  primary  area  of  study?  (if  applicable)  [free  form  text  field] 

12.5.  What  do  you  consider  to  be  your  two  greatest  academic  strength  areas?  (e.g.,  math,  science, 
computers,  English,  art,  etc.)  [free  form  memo  field] 

12.6.  What  types  of  extracurricular  activities,  if  any,  are  you  currently  engaged  in?  (e.g.,  soccer, 
chess  club,  student  government,  computer  club,  etc.)  [free  form  memo  field] 

13.  Cyber  Security  Proficiency 

13.1.  Have  you  ever  had  formal  training/  coursework  on  cyber  security  topics?  [yes/  no  selection] 

13.2.  On  a  scale  of  1-10,  what  is  your  level  of  familiarity/  comfort  with  the  field  of  cyber  security? 
[drop  down  menu] 

13.2.1.  1,  No  Familiarity 

13.2.2.  2 

13.2.3.  3,  Novice 

13.2.4.  4 

13.2.5.  5,  Basic  Familiarity 

13.2.6.  6 

13.2.7.  7,  Proficient 

13.2.8.  8 

13.2.9.  9,  Highly  Proficient 
13.2.10.10,  Expert 

13.3.  If  you  have  a  background  in  cyber  security,  what  areas  of  cyber  security  do  you  consider 
yourself  most  proficient  in?  (e.g.,  system  administration,  programming,  forensics,  system 
penetration,  etc.)  [free  form  memo  field] 

14.  Competition/  Event  History 

14.1.  Where  did  you  first  hear  about  the  USCC  organization?  [drop  down  menu  w/  other  option] 
14.  L  L  Email/  Newsletter 

14. 1.2.  Facebook 

14. 1.3.  Family  or  Friend 

14. 1.4.  Magazine  Article 

14. 1.5.  Newspaper  Story 

14. 1.6.  Professor 

14. 1.7.  TV  News 

14. 1.8.  Twitter 

14. 1.9.  University  Publication 
14.  L 10. Website/  Search  Engine 
14.  LIE  YouTube 

14.2.  Have  you  engaged  in  cyber/  security  activities,  events,  or  competitions  in  the  past?  [yes/  no 
selection] 

14.3.  If  you  have  engaged  in  similar  events  in  the  past,  briefly  list  these  previous  events  and 
activities,  [free  form  memo  field] 

15.  Professional  Interests 

15.1.  What  is  your  desired/  planned  career  field?  [free  form  text  field] 

15.2.  Have  you  previously  considered  a  career  in  cyber  security?  [yes/  no  selection] 

15.3.  On  a  scale  of  1-5,  after  your  participation  in  this  event  is  what  is  the  likelihood  you  will 
to  explore  cyber  security  as  a  career  path?  [drop  down  menu] 

15.3.1.  1,  Not  At  All 

15.3.2.  2,  Unlikely 
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15.3.3.  3,  Likely 

15.3.4.  4,  Very  Likely 

15.3.5.  5,  Certainly 

16.  Closing 

16.1.  If  you  have  any  additional  comments  or  feedback,  please  provide  it  here,  [free  form  memo 

field] 

Each  day  of  each  camp  used  a  new  instance  of  the  same  electronic  survey  for  collection  which  was 
differentiated  by  the  course  selected  in  2.1  above.  The  table  below  provides  an  overview  of  the  day 
specific  survey  options  for  the  2011  camps.  The  2011  camps  included; 

•  California  State  Polytechnic  University,  Ponoma  (California)  Camp  from  July  1 1  to  July  15,  201 1 

•  Community  College  of  Baltimore  County,  Essex  (Maryland)  Camp  from  July  1 1  to  July  15,  201 1 

•  University  of  Missouri,  Columbia  (Missouri)  Camp  from  July  25  to  July  29,  2011 

•  J.  Sargeant  Reynolds  Community  College,  Richmond  (Virginia)  Camp  from  August  1  to  August 
5,2011 

•  Delaware  Technical  and  Community  College,  Dover  (Delaware)  Camp  from  August  8  to  August 


12, 2010 


Location 

Day 

Date 

Course 

Instructor 

California 

One 

August  1 1 

Web  Application  Penetration  Testing 

Kevin  Johnson 

California 

Two 

August  12 

System  Hardening  &  Detection 

Frank  DiMaggio 

California 

Three 

August  13 

Forensics 

Mike  Murr 

California 

Four 

August  14 

Reverse  Engineering  Malware 

Hal  Pomeranz 

California 

Five 

August  15 

Capture  the  Flag 

Maryland 

Operating  Systems 

Maryland 

System  Administration 

Maryland 

Networking 

Maryland 

Programming 

Maryland 

Site  Visit 

Maryland 

Five 

July  15 

Cyber  Defense  Exercise 

Missouri 

One 

July  25 

Web  Application  Penetration  Testing 

Justin  Searle 

Missouri 

Two 

July  26 

System  Hardening  &  Detection 

Justin  Searle 

Missouri 

Three 

July  27 

Packet  Crafting  with  Scapy 

Troy  Jordan 

Missouri 

Four 

July  28 

Wireless  &  Bluetooth  Security 

John  Paul  Dunning 

Missouri 

Five 

July  29 

Capture  the  Flag 

Virginia 

One 

August  1 

Packet  Crafting  with  Scapy 

Judy  Novak 

Virginia 

Two 

August  2 

Wireless  &  Bluetooth  Security 

John  Paul  Dunning 

Virginia 

Three 

August  3 

Forensics 

Mike  Murr 

Virginia 

Four 

August  4 

Reverse  Engineering  Malware 

Hal  Pomeranz 

Virginia 

Five 

August  5 

Capture  the  Flag 

Delaware 

One 

August  8 

General  Penetration  Testing 

Eric  Amoth 

Delaware 

Two 

August  9 

Web  Application  Penetration  Testing 

Justin  Searle 

Delaware 

Three 

August  10 

System  Hardening  &  Detection 

Justin  Searle 

Delaware 

Four 

August  1 1 

Forensics 

Mike  Murr 

Delaware 

Five 

August  12 

Capture  the  Flag 
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2011  CTF  Survey  Structure 

1.  Event  Setup 

1.1.  Was  the  vendor  helpful  in  explaining  the  set  up  for  this  event?  [yes/  no  selection  w/  detail  memo 
field  option] 

1.2.  On  a  scale  of  1-5,  how  would  you  rate  the  quality  of  the  technical  documentation  provided  by  the 
vendor?  [drop  down  menu  w/  detail  memo  field  option] 

1.2. 1.1,  Inadequate 

1.2.2. 2,  Poor 

1.2. 3. 3,  Acceptable 

1. 2.4.4,  Good 

1.2. 5. 5,  Excellent 

1.3.  Did  the  competition  appear  to  provide  the  technical  challenge  for  the  class?  [yes/  no  selection  w/ 
detail  memo  field  option] 

1 .4.  Do  you  feel  the  technical  set  up  was  easy  to  achieve?  [yes/  no  selection] 

1.5.  Did  the  technical  performance  meet  your  expectations?  [yes/  no  selection] 

2.  Evaluation 

2.1.  On  a  scale  of  1-10,  what  is  your  overall  evaluation  of  this  course/  event?  [drop  down  menu] 

2.1. 1.1,  Bad 

2. 1.2. 2, 

2. 1.3. 3,  Poor 

2. 1.4.4, 

2. 1.5. 5,  Marginal 

2. 1.6.6, 

2. 1.7.7,  Good 

2. 1.8. 8, 

2. 1.9.9,  Great 

2.1.10.  10,  Excellent 

2.2.  On  a  scale  of  1-10,  what  is  your  evaluation  of  the  scoreboard  display?  [drop  down  menu] 

2.2.1. 1,  Bad 

2.2.2. 2, 

2.2. 3. 3,  Poor 

2.2.4. 4, 

2.2. 5. 5,  Marginal 

2.2.6. 6, 

2.2.7. 7,  Good 

2.2.8. 8, 

2.2.9. 9,  Great 

2.2.10.  10,  Excellent 

2.3.  On  a  scale  of  1-10,  what  is  your  evaluation  of  the  network  display  for  the  TAs?  [drop  down 
menu] 

2.3. 1.1,  Bad 

2.3.2. 2, 

2. 3. 3. 3,  Poor 

2. 3.4.4, 

2. 3. 5. 5,  Marginal 

2. 3. 6. 6, 
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2.3. 7. 7,  Good 

2.3. 8. 8, 

2. 3. 9. 9,  Great 

2.3.10.  10,  Excellent 

2.4.  On  a  scale  of  1-10,  what  is  your  evaluation  of  the  ease  of  use  for  the  technical  solution?  [drop 
down  menu] 

2.4. 1.1,  Bad 

2.4.2. 2, 

2.4.3. 3,  Poor 

2.4.4. 4, 

2. 4. 5. 5,  Marginal 

2.4.6. 6, 

2.4.7. 7,  Good 

2.4.8. 8, 

2.4.9. 9,  Great 

2.4.10.  10,  Excellent 

3.  Event  Assessment 

3.1.  Would  you  use  this  solution  for  your  upcoming  activities?  [free  form  memo  field] 

3.2.  What  are  the  strengths  of  this  competition?  [free  form  memo  field] 

3.3.  What  are  the  areas  for  improvements  for  this  event/  competition?  [free  form  memo  field] 

3.4.  Would  you  recommend  this  event/  competition  to  your  peers?  [yes/  no  selection] 

4.  Closing 

4.1.  If  you  have  any  additional  comments  or  feedback,  please  provide  it  here,  [free  form  memo  field] 

201 0  Pilot  Survey  Structures 

In  2010  the  pilot  camps  utilized  hardcopy  surveys  specific  to  each  day  of  the  camp  to  collect  the 
necessary  feedback  from  participants. 

2010  Pilot  Camp  Survey  Structure 

1.  Feedback 

1.1.  Overall  Course  Evaluation  [numeric  scale,  single  selection] 

1.1.1.  Scale  1  -  10,  1  poor  and  10  excellent 

1.2.  [ Teachers  Name ]  Teaching  Skill  [numeric  scale,  single  selection] 

1.2.1. Scale  1  -  10,  1  poor  and  10  excellent 

1.3.  Value  of  Course  Content  [numeric  scale,  single  selection] 

1.3.1. Scale  1  -  10,  1  poor  and  10  excellent 

1 .4.  If  you  liked  the  course  and  wouldn’t  mind  sharing  your  comments  with  others  who  might 
consider  taking  this  course  in  the  future,  please  write  a  sentence  about  why  you  feel  this  course 
was  valuable,  [free  form  essay  field] 

1.5.  What  specific  recommendations  would  you  offer  to  make  this  course  more  valuable  to  you?  [free 
form  essay  field] 

1.6.  Name  [free  form  text  field] 

1.7.  College  [free  form  text  field] 
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Each  day  of  each  camp  had  a  separate  hardcopy  survey  for  collection  which  varied  only  in  the  Teacher’s 
Name  used  in  1.2  above.  The  table  below  provides  an  overview  of  the  day  specific  survey  information  for 
the  2010  camps.  The  2010  camps  included; 

•  California  State  Polytechnic  University,  Ponoma  (California)  Camp  from  July  19  to  July  23,  2010 

•  Polytechnic  Institute  of  NYU  (New  York)  Camp  from  July  26  to  July  29,  2010 

•  Wilmington  University  (Delaware)  Camp  from  August  9  to  August  12,  2010 


Location 

Day 

Date 

Course 

Instructor 

California 

One 

July  19 

Developing  Exploits  for  Penetration 
Testers  and  Security 

Stephen  Sims 

California 

Two 

July  20 

Linux/  Unix  Security 

Hal  Pomeranz 

California 

Three 

July  21 

Incident  Handling/  Hacking  Techniques 

James  Shewmaker 

California 

Four 

July  22 

Forensic  Intensive 

Mike  Murr 

New  York 

One 

July  26 

Reverse  Engineering  Malware:  Malware 
Analysis  and  Techniques 

Lenny  Zeltser 

New  York 

Two 

July  27 

Windows  Forensics 

Rob  Lee 

New  York 

Three 

July  28 

Incident  Handling/  Hacking  Techniques 

Dr  Johannes  Ullrich 

New  York 

Four 

July  29 

Power  Packet  Crafting  with  Scapy 

Judy  Novak 

Delaware 

One 

August  9 

Windows  Forensics 

Rob  Lee 

Delaware 

Two 

August  10 

Incident  Handling/  Hacking  Techniques 

Marcus  Sachs 

Delaware 

Three 

August  1 1 

Network  Penetration  Testing  and  Ethical 
Hacking 

Ed  Skoudis 

Delaware 

Four 

August  12 

Power  Packet  Crafting  with  Scapy 

Judy  Novak 
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Comprehensive  2011  Participant  Survey  Results  Analysis 


Data  Preparation 

The  data  from  each  camp  was  collected  separately  via  camp  specific  SurveyMonkey  surveys,  managed  in 
Excel  format,  and  reviewed  for  completeness  and  accuracy.  Records  in  the  data  which  had  any  of  the 
following  errors  or  inconsistencies  were  removed  from  the  dataset  prior  to  analysis. 

1 .  Out  of  scope  start  date  or  end  date.  Surveys  taken  significantly  prior  to  or  significantly  after  the 
camp  were  discarded. 

2.  Invalid  data.  Surveys  with  unrealistic  names,  invalid  email  format  or  filled  with  garbage  data 
(e.g.,  all  fields  with  ‘ddddd’)  were  discarded. 

3.  Incomplete  surveys.  Surveys  in  which  the  participant  only  answered  name  and  email  questions,  or 
less,  but  did  not  answer  any  of  the  material  survey  questions  were  discarded. 

In  the  majority  of  cases,  removed  records  had  more  than  one  of  the  above  data  quality  and /  or 
completeness  issues  and  in  some  cases  the  removed  surveys  were  clearly  test  runs  or  partial  completions 
by  camp  staff  and  administrators. 

Below  is  a  summary  of  the  records  removed  prior  to  analysis. 

1 .  CA  Camp  Records  Removed  (1  Total  Record  Removed) 

1.1.  Respondent  ID:  1481534734  removed  due  to  incompleteness. 

2.  MD  Camp  Records  Removed  (1  Total  Record  Removed) 

2.1.  Respondent  ID:  1479603532  removed  due  to  invalid  name  and  email. 

3.  MO  Camp  Records  Removed  (3  Total  Records  Removed) 

3.1.  Respondent  ID:  1496098208  removed  due  to  incompleteness. 

3.2.  Respondent  ID:  1494073843  removed  due  to  incompleteness. 

3.3.  Respondent  ID:  1493653669  removed  due  to  incompleteness. 

4.  VA  Camp  Records  Removed  (6  Total  Records  Removed) 

4.1.  Respondent  ID:  1499000816  removed  due  to  out  of  scope  start  date. 

4.2.  Respondent  ID:  1509326291  removed  due  to  incompleteness. 

4.3.  Respondent  ID:  1502520927  removed  due  to  incompleteness. 

4.4.  Respondent  ID:  1502151801  removed  due  to  invalid  name  and  email. 

4.5.  Respondent  ID:  1502275647  removed  due  to  incompleteness. 

4.6.  Respondent  ID:  1502153554  removed  due  to  incompleteness. 

5.  DE  Camp  Records  Removed  (2  Total  Records  Removed) 

5.1.  Respondent  ID:  1509976005  removed  due  to  out  of  scope  start  date. 

5.2.  Respondent  ID:  1513799112  removed  due  to  incompleteness. 

Following  the  removal  of  the  above  records,  the  remaining  data  was  divided  into  three  separate  data  sets 
for  each  camp.  These  data  sets  included  course  specific  questions,  individual  participant  specific 
questions  and  questions  not  conducive  to  analysis. 

The  first  data  set  contained  responses  to  questions  specific  to  the  courses  within  the  camp.  This  data  was 
separated  out  by  camp  and  then  further  divided  out  by  specific  day  of  the  camp.  These  responses  provide 
insight  into  the  specific  instructors,  courses  and  curriculum  for  each  day  of  the  camps.  This  first  data  set 
included  responses  for  the  following  questions; 
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•  Question  3:  What  USCC  event  did  you  attend? 

•  Question  6:  On  a  scale  of  1-10,  what  is  your  overall  evaluation  of  this  course/  event? 

•  Question  7:  On  a  scale  of  1-10,  what  is  your  overall  evaluation  of  the  instructor's  teaching  skill? 

•  Question  8:  On  a  scale  of  1-10,  what  is  your  overall  evaluation  of  the  value  of  the  course  content? 

The  second  data  set  included  the  questions  to  which  responses  could  not  effectively  be  analyzed  or 
summarized.  These  responses  were  not  conducive  to  significant  analysis  and  are  not  included  in  this 
document. 

•  Question  1 :  What  is  your  first  and  last  name? 

•  Question  2:  What  is  your  email  address? 

•  Question  12:  What  is  your  primary  city  of  residence? 

•  Question  20:  What  types  of  extracurricular  activities,  if  any,  are  you  currently  engaged  in?  (e.g., 
soccer,  chess  club,  student  government,  computer  club,  etc.) 

•  Question  30:  If  you  have  any  additional  comments  or  feedback,  please  provide  it  here. 

The  final  data  set  included  responses  which  provided  insight  on  the  individual  participant.  This  included 
all  the  questions  not  in  the  above  two  data  sets.  The  data  for  this  data  set  was  filtered  to  ensure  each 
participant  was  counted  only  one  time.  Specifically,  many  participants  completed  multiple  instances  of 
the  survey,  one  for  each  day  of  the  camp.  For  example,  if  a  participant  completed  five  instances  of  the  CA 
survey,  one  for  each  day  of  the  camp,  then  all  five  responses  would  be  included  in  the  first  data  set  to 
reflect  the  specific  responses  for  each  individual  course/  day  of  the  camp.  However,  those  records 
(identified  by  a  name  and/  or  email  match)  would  be  consolidated  into  a  single  record  for  the  remaining 
analysis  in  this  final  data  set.  This  consolidation  of  records  for  analysis  in  this  final  data  set  ensures 
numbers  are  not  artificially  inflated  in  the  analysis  of  these  questions.  For  example,  without  consolidation 
but  accounting  for  participants  completing  multiple  instances  of  the  survey  for  multiple  days,  the  analysis 
results  could  show  82  male  responses  and  36  female  responses  for  a  camp  which  only  had  30  total 
participants.  For  discrepancies  in  data  provided  by  individuals  across  multiple  responses,  e.g.,  different 
age  entries  with  the  same  name  and/  or  email,  the  first  response  provided  (as  determined  by  the  ‘Start 
Date’  field)  was  used  for  analysis. 
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Summary:  Total  Survey  Responses 


O 

> 

MD 

MO 

VA 

DE 

Total 

Number 

108 

■■ 

23 

121 

25 

83 

360 

Percent 

30.0% 

6.4% 

33.6% 

6.9% 

23.1% 

100% 

Summary:  Total  Distinct  Participants  (Unique  Name  and/  or  Email  Address) 
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Summary:  Total  Responses  per  Individual 


0  20  40  60  80 

_ _ _ _ _ / 


CA 

MD 

MO 

VA 

DE 

Completed  1  Survey 

6 

23 

4 

23 

■■ 

18 

Completed  2  Survey 

8 

0 

6 

1 

4 

Completed  3  Survey 

10 

■■ 

0 

9 

0 

■■ 

4 

Completed  4  Survey 

14 

0 

11 

0 

0 

Completed  5  Survey 

0 

■■ 

0 

7 

0 

mm 

9 
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Question  3:  What  USCC  event  did  you  attend? 
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Question  4:  Is  this  your  first  USCC  event? 


CA 

MD 

MO 

VA 

DE 

Total 

Yes  Responses 

33 

21 

34 

24 

30 

142 

Yes  Percentage 

86.84% 

91.30% 

91.89% 

100.00% 

88.24% 

91.03% 

No  Responses 

5 

2 

3 

0 

4 

■■ 

14 

No  Percentage 

13.16% 

8.70% 

8.11% 

0.00% 

11.76% 

8.97% 

No  (Blank)  Response 

0 

0 

0 

0 

1 

i 
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Question  5:  Where  did  you  first  hear  about  this  event? 

■  Email/ Newsletter 

■  Facebook 

■  Family  or  Friend 

■  Magazine  Article 

■  Newspaper  Story 

■  Professor 

■  TV  News 

■  Twitter 

■  University  Publication 

■  Website/  Search  Engine 

■  YouTube 


CA 

■■ 

MD 

■■ 

MO 

VA 

■■ 

DE 

■■ 

Total 

Percent 

1 

Email/  Newsletter 

6 

2 

■■ 

12 

7 

■■ 

5 

32 

21.48% 

Facebook 

0 

1 

0 

0 

0 

1 

0.67% 

Family  or  Friend 

6 

4 

■■ 

3 

3 

■■ 

4 

20 

13.42% 

Magazine  Article 

0 

0 

0 

1 

0 

1 

0.67% 

Newspaper  Story 

0 

■■ 

0 

■■ 

0 

0 

■■ 

0 

0 

0.00% 

Professor 

22 

10 

14 

2 

17 

65 

43.62% 

TV  News 

0 

0 

0 

0 

0 

0 

0.00% 

Twitter 

0 

0 

1 

2 

0 

3 

2.01% 

University  Publication 

1 

3 

■■ 

2 

1 

■■ 

7 

14 

9.40% 

Website/  Search  Engine 

2 

1 

3 

6 

1 

13 

8.72% 

YouTube 

0 

■■ 

0 

0 

0 

■■ 

0 

0 

0.00% 

No  Response  or  Other 

6 

2 

12 

7 

5 

32 

21.48% 
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Question  6,7  &  8:  California  Camp  Responses 


25 

20 

15 

10 


L 

■L.  .  . 

t  J 

a 


07 


I CTF 

17/11 

17/12 

17/13 

17/14 


Question6:  On  a  scale  of  1-10,  what  is  your  overall  evaluation  of  this  course/  event? 


Question  7:  On  a  scale  of  1-10,  what  is  your  overall  evaluation  of  the  instructor's  teaching  skill? 


Camp  -  Course 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

Tota 

l 

CA  -  Capture  the  Flag  (CTF) 

0 

0 

0 

0 

0 

0 

0 

0 

l 

0 

1 

CA  -  7/11/2011,  Kevin  Johnson,  WebAppPenTest 

0 

0 

0 

0 

l 

0 

0 

0 

9 

23 

33 

CA  -  7/12/2011,  Frank  DiMaggio,  SystemHardening  &  Detection 

0 

0 

0 

0 

i 

0 

0 

0 

6 

20 

27 

CA  -  7/13/2011,  Mike  Murr,  Forensics 

0 

0 

0 

0 

0 

0 

3 

1 

7 

8 

19 

CA  -  7/14/2011,  Hal  Pomeranz,  Reverse  Engineering  Malware 

0 

0 

0 

0 

0 

0 

3 

2 

5 

18 

28 

Question8:  On  a  scale  of  1-10,  what  is  your  overall  evaluation  of  the  instructor's  teaching  skill? 
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Question  6,  7  &  8:  Maryland  Camp  Responses 


Question6:  On  a  scale  of  1-10 ,  what  is  your  overall  evaluation  of  this  course/  event? 


Camp  -  Course 


10 


Total 


MD  -  Operating  Systems 

0 

0 

II 

0 

0 

0 

0 

2  1 

II 

0 

II 

1 

II 

4 

MD  -  System  Administration 

0 

0 

0 

0 

0 

2 

0  0 

0 

0 

2 

MD  -  Networking 

0 

0 

II 

0 

0 

0 

0 

0  1 

II 

1 

1 

II 

3 

MD  -  Programming 

0 

0 

0 

0 

0 

0 

0  0 

0 

0 

0 

MD  -  Cyber  Defense  Exercise 

1 

0 

II 

0 

0 

1 

0 

3  1 

II 

3 

II 

5 

ll 

14 

Question  7:  On  a  scale  of  1-10 ,  what  is  your  overall  evaluation  of  the  instructor 1 

rs  teaching  skill? 

Camp  -  Course 

111 

2 

II 

3 

II 

41 

5 

II 

6 

II 

7||8 

II 

9 

II 

10 

II 

Total 

MD  -  Operating  Systems 

0 

0 

II 

0 

0 

0 

II 

0 

i  i 

II 

1 

1 

II 

4 

MD  -  System  Administration 

0 

0 

0 

0 

1 

0 

0  0 

0 

1 

2 

MD  -  Networking 

0 

0 

II 

0 

0 

0 

0 

1  0 

II 

1 

1 

II 

3 

MD  -  Programming 

0 

0 

0 

0 

0 

0 

0  0 

0 

0 

0 

MD  -  Cyber  Defense  Exercise 

1 

0 

II 

0 

0 

0 

0 

0  1 

II 

4 

8 

Ml 

14 

Question 8:  On  a  scale  of  1-10 ,  what  is  your  overall  evaluation  of  the  instructor's  teaching  skill? 

Camp  -  Course 

1 

2 

II 

3 

II 

41 

5 

II 

6 

II 

7||8 

II 

9 

II 

10 

ll 

Total 

MD  -  Operating  Systems 

0 

0 

II 

0 

0 

0 

II 

0 

i  i 

II 

0 

2 

4 

MD  -  System  Administration 

0 

0 

0 

0 

0 

1 

0  1 

0 

0 

2 

MD  -  Networking 

0 

0 

0 

0 

0 

0 

0  1 

II 

0 

2 

3 

MD  -  Programming 

0 

0 

0 

0 

0 

0 

0  0 

0 

0 

0 

MD  -  Cyber  Defense  Exercise 

1 

0 

II 

0 

1 

0 

0 

II 

4  0 

II 

1 

7 

II 

14 

*MD  -  Site  Visit  and  MD  -  Guest  Speaker  result  rows  removed  because  there  was  no  response. 
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Question  6,  7  &  8:  Missouri  Camp  Responses 


Question6:  On  a  scale  of  1-10,  what  is  your  overall  evaluation  of  this  course/  event? 


Camp  -  Course 

1  2 

3 

4 

5 

6 

7 

8 

9 

10 

Total 

MO  -  7/25  -  Wep  App  Pen  Test  -  Justin  Searle 

0  0 

II 

0 

ll 

o 

0 

1 

3 

6 

11 

11 

32 

MO  -  7/26  -  System  Hardening  &  Detection  -  Justin  Searle 

0  0 

0 

0 

1 

0 

10 

4 

8 

3 

26 

MO  -  7/27  -  Packet  Crafting  with  Scapy  -  Troy  Jordan 

0  0 

II 

0 

1 

0 

1 

3 

10 

8 

5 

28 

MO  -  7/28  -  Wireless  &  Bluetooth  Security  -  John  Paul  Dunning 

0  0 

0 

0 

0 

0 

3 

2 

1 

3 

9 

MO  -  7/29  -  Capture  the  Flag 

1  0 

II 

2 

0 

3 

3 

6 

5 

5 

1 

26 

Question  7:  On  a  scale  of  1-10 ,  what  is  your  overall  evaluation  of  the  instructor's  teaching  skill? 

Camp  -  Course 

1  2 

3 

4 

5 

6 

7 

8 

9 

10 

Total 

MO  -  7/25  -  Wep  App  Pen  Test  -  Justin  Searle 

0  0 

II 

0 

o 

0 

0 

o 

1 

14 

17 

32 

MO  -  7/26  -  System  Hardening  &  Detection  -  Justin  Searle 

0  0 

0 

0 

0 

0 

2 

4 

10 

10 

26 

MO  -  7/27  -  Packet  Crafting  with  Scapy  -  Troy  Jordan 

0  0 

0 

0 

1 

1 

2 

8 

11 

5 

28 

MO  -  7/28  -  Wireless  &  Bluetooth  Security  -  John  Paul  Dunning 

0  0 

0 

0 

0 

0 

4 

1 

1 

3 

9 

MO  -  7/29  -  Capture  the  Flag 

1  1 

3 

1 

3 

1 

2 

2 

5 

3 

22 

Question8:  On  a  scale  of  1-10,  what  is  your  overall  evaluation  of  the  instructor's  teaching  skill? 


Camp  -  Course 

i 

2 

3 

4 

5 

6 

7 

8 

9 

10 

Total 

MO  -  7/25  -  Wep  App  Pen  Test  -  Justin  Searle 

0 

0 

0 

o 

0 

0 

5 

6 

9 

12 

32 

MO  -  7/26  -  System  Hardening  &  Detection  -  Justin  Searle 

0 

0 

0 

0 

1 

0 

8 

6 

6 

4 

25 

MO  -  7/27  -  Packet  Crafting  with  Scapy  -  Troy  Jordan 

0 

0 

1 

0 

0 

II 

0 

4 

9 

6 

8 

28 

MO  -  7/28  -  Wireless  &  Bluetooth  Security  -  John  Paul  Dunning 

0 

0 

0 

0 

0 

0 

3 

2 

1 

3 

9 

MO  -  7/29  -  Capture  the  Flag 

1 

0 

II 

0 

1 

4 

II 

1 

II 

5 

5 

4 

3 

24 
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Question  6,  7  &  8:  Virginia  Camp  Responses 


Question6:  On  a  scale  of  1-10,  what  is  your  overall  evaluation  of  this  course/  event? 


Camp  -  Course 

1  2  ||  3  4  H  5  ||  6  7|[8||9  10  Total 

VA  -  8/1  -  Packet  Crafting  with  Scapy  -  Judy  Novak 

0  0  0  0  1  0  4  1  7  8  21 

VA  -  8/2  -  Wireless  and  Bluetooth  Security  -  John  Paul  Dunning 

0000000100  1 

VA  -  8/3  -  Forensics  -  Mike  Murr  -  also  taught  at  the  CA  Camp 

0  0  0  0  0  0  0  0  2  1  3 

VA  -  8/4  -  Reverse  Engineering  Malware  -  Hal  Pomeranz 

0000000000  0 

Question  7:  On  a  scale  of  1-10 ,  what  is  your  overall  evaluation  of  the  instructor's  teaching  skill? 

Camp  -  Course 

1  2  II  3  4||  5  II  6  7||  8  II  9  10  Total 

VA  -  8/1  -  Packet  Crafting  with  Scapy  -  Judy  Novak 

0  0  0  0  0  0  2  4  7  8  21 

VA  -  8/2  -  Wireless  and  Bluetooth  Security  -  John  Paul  Dunning 

0000000010  1 

VA  -  8/3  -  Forensics  -  Mike  Murr  -  also  taught  at  the  CA  Camp 

0  0  0  0  0  0  0  0  1  2  3 

VA  -  8/4  -  Reverse  Engineering  Malware  -  Hal  Pomeranz 

0000000000  0 

Question 8:  On  a  scale  of  1-10,  what  is  your  overall  evaluation  of  the  instructor's  teaching  skill? 

Camp  -  Course 

1  2  II  3  4||  5  II  6  7||  8  II  9  10  Total 

VA  -  8/1  -  Packet  Crafting  with  Scapy  -  Judy  Novak 

0  0  0  0  0  0  3  1  6  11  21 

VA  -  8/2  -  Wireless  and  Bluetooth  Security  -  John  Paul  Dunning 

0000000100  1 

VA  -  8/3  -  Forensics  -  Mike  Murr  -  also  taught  at  the  CA  Camp 

0  0  0  0  0  0  0  0  2  1  3 

VA  -  8/4  -  Reverse  Engineering  Malware  -  Hal  Pomeranz 

0000000000  0 
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Question  6,  7  &  8:  Delaware  Camp  Responses 
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Q8 

■  Mon 

■  Tue 

■  Wed 

■  Thr 

■  CTF 


Question6:  On  a  scale  of  1-10,  what  is  your  overall  evaluation  of  this  course/  event? 


Camp  -  Course 

i 

2 

3 

4 

5 

6 

7 

8 

9 

10 

Total 

DE  -  Monday  -  General  Penetration  Testing  -  Eric  Arnoth 

0 

0 

II 

0 

II 

o 

1 

1 

2 

2 

7 

9 

22 

DE  -  Tuesday  -  Web  App  Pen  Testing  -  Justin  Searles 

0 

0 

0 

0 

0 

0 

1 

2 

5 

7 

15 

DE  -  Wednesday  -  System  Hardening  and  Detection  -  Justin  Searles 

0 

0 

II 

0 

0 

0 

0 

1 

2 

2 

4 

9 

DE  -  Thursday  -  Forensics  -  Mike  Murr 

0 

0 

0 

0 

0 

1 

2 

2 

3 

4 

12 

DE  -  Capture  the  Flag 

0 

0 

0 

1 

3 

0 

2 

3 

7 

7 

23 

Question  7:  On  a  scale  of  1-10,  what  is  your  overall  evaluation  of  the  instructor's  teaching  skill? 


Camp  -  Course 

i 

2 

3 

4 

5 

6 

7 

8 

9 

10 

Total 

DE  -  Monday  -  General  Penetration  Testing  -  Eric  Arnoth 

0 

0 

ll 

0 

o 

0 

0 

2 

1 

II 

10 

ll 

9 

22 

DE  -  Tuesday  -  Web  App  Pen  Testing  -  Justin  Searles 

0 

0 

0 

0 

0 

0 

2 

2 

3 

8 

15 

DE  -  Wednesday  -  System  Hardening  and  Detection  -  Justin  Searles 

0 

0 

II 

0 

0 

0 

0 

2 

i 

II 

2 

II 

4 

9 

DE  -  Thursday  -  Forensics  -  Mike  Murr 

0 

0 

0 

0 

1 

0 

2 

1 

2 

6 

12 

DE  -  Capture  the  Flag 

0 

0 

1 

0 

1 

0 

3 

3 

II 

7 

7 

22 

Question8:  On  a  scale  of  1-10,  what  is  your  overall  evaluation  of  the  instructor's  teaching  skill? 


Camp  -  Course 

i 

2 

3 

4 

5 

6 

7 

8 

9 

10 

Total 

DE  -  Monday  -  General  Penetration  Testing  -  Eric  Arnoth 

0 

0 

0 

o 

0 

0 

3 

2 

7 

10 

22 

DE  -  Tuesday  -  Web  App  Pen  Testing  -  Justin  Searles 

0 

0 

0 

0 

0 

1 

1 

1 

5 

7 

15 

DE  -  Wednesday  -  System  Hardening  and  Detection  -  Justin  Searles 

0 

0 

0 

0 

0 

0 

2 

o 

3 

4 

9 

DE  -  Thursday  -  Forensics  -  Mike  Murr 

0 

0 

0 

0 

0 

0 

1 

3 

2 

6 

12 

DE  -  Capture  the  Flag 

0 

0 

II 

0 

1 

0 

II 

1 

II 

5 

2 

6 

7 

22 
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Question  9:  What  encouraged  you  to  participate  in  this  event? 


CA 

MD 

MO 

VA 

DE 

Total 

Percent 

Being  invited/  exclusivity 

0 

o 

0 

i 

3 

4 

2.41% 

Camp  cost  (free) 

2 

2 

5 

4 

0 

13 

7.83% 

Camp  location 

i 

0 

0 

0 

o 

i 

0.60% 

Camp,  sponsor  and 
organization  reputations 

1 

0 

1 

7 

2 

11 

6.63% 

Career  opportunities 

7 

i 

3 

i 

4 

16 

9.64% 

Competition  preparation/ 

CTF 

0 

1 

0 

1 

2 

4 

2.41% 

Event  coverage  and  publicity 

o 

0 

o 

1 

i 

0.60% 

Experience  in  related  events/ 
activities 

2 

0 

0 

0 

0 

2 

1.20% 

Exploring  something  new 

0 

2 

1 

0 

i 

4 

2.41% 

External  pressure  to  attend 

0 

2 

0 

0 

0 

2 

1.20% 

Interest  in  the  cyber  security 
field 

7 

1 

1 

3 

3 

15 

9.04% 

Opportunity  to  leverage 
course  material 

0 

1 

0 

0 

0 

1 

0.60% 

Opportunity  to  learn  new 
things  and/  or  develop  skills 

10 

9 

13 

7 

8 

47 

28.31% 

Personal  or  academic 
challenge 

0 

0 

3 

0 

0 

3 

1.81% 

Personal  passion  for 
computers 

3 

1 

0 

0 

4 

8 

4.82% 

Professional  networking 

3 

0 

2 

3 

0 

8 

4.82% 

Recommendation  of  a 
professor 

2 

2 

1 

1 

5 

11 

6.63% 

Recommendation  of  a  friend 
or  family 

2 

3 

0 

1 

2 

8 

4.82% 

Reputation  of  the  instructors 

4 

0 

i 

2 

0 

7 

4.22% 

Scholarship  opportunities 

0 

1 

0 

0 

0 

1 

0.60% 

*N limbers  for  like  terms  have  been  consolidated.  All  terms  in  responses  with  multiple  terms  are  included. 
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Question  10:  What  is  your  age? 


ICA 
IMD 
i  MO 
IVA 
IDE 


CA 

MD 

MO 

VA 

DE 

Total 

Percent 

Less  than  15 

0 

1 

■■ 

1 

0 

0 

2 

1.27% 

15 

0 

4 

0 

0 

0 

4 

2.55% 

16 

0 

7 

■■ 

0 

0 

0 

7 

4.46% 

17 

0 

5 

0 

0 

0 

5 

3.18% 

18 

3 

3 

■■ 

1 

0 

1 

8 

■■ 

5.10% 

19 

3 

0 

0 

1 

1 

5 

3.18% 

20 

2 

0 

■■ 

1 

2 

0 

5 

■■ 

3.18% 

21 

3 

0 

2 

2 

7 

14 

8.92% 

22 

3 

0 

■■ 

5 

3 

2 

13 

8.28% 

23 

0 

0 

3 

1 

3 

7 

4.46% 

24 

2 

0 

■■ 

0 

1 

0 

3 

■■ 

1.91% 

25 

2 

1 

4 

1 

3 

11 

7.01% 

26 

4 

0 

■■ 

1 

1 

3 

9 

■■ 

5.73% 

27 

3 

0 

0 

2 

0 

5 

3.18% 

28 

1 

0 

5 

1 

2 

9 

5.73% 

29 

2 

0 

0 

1 

1 

4 

2.55% 

30 

1 

0 

■■ 

2 

0 

4 

7 

■■ 

4.46% 

31 

1 

0 

1 

0 

1 

3 

1.91% 

32 

1 

0 

■■ 

0 

0 

1 

2 

■■ 

1.27% 

33 

1 

0 

0 

1 

1 

3 

1.91% 

34 

0 

0 

■■ 

2 

1 

1 

4 

2.55% 

35 

0 

0 

0 

0 

0 

0 

0.00% 

36 

0 

0 

0 

1 

0 

1 

0.64% 

37 

0 

0 

1 

1 

0 

2 

1.27% 

38 

0 

0 

■■ 

1 

1 

0 

2 

■■ 

1.27% 

39 

1 

0 

0 

0 

0 

1 

0.64% 

40 

0 

0 

0 

0 

0 

0 

0.00% 

Greater  than  40 

4 

1 

5 

2 

2 

14 

8.92% 

No  Response 

1 

1 

2 

1 

2 

7 

■■ 

4.46% 

Minimum  age  for  overnight  camps  was  18. 
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Question  11:  What  is  your  gender? 


HI 

CA 

■■ 

MD 

■I 

MO 

VA 

DE 

■I 

Total 

■I 

Percent 

Male  Responses 

HI 

33 

21 

33 

23 

29 

139 

88.54% 

Male  Percentage 

86.84% 

91.30% 

89.19% 

95.83% 

82.86% 

88.54% 

___ 

Female  Responses 

5 

2 

3 

0 

5 

15 

9.55% 

Female  Percentage 

13.16% 

8.70% 

8.11% 

0.00% 

14.29% 

9.55% 

___ 

No  Response 

0 

0 

i 

1 

1 

3 

■I 

1.91% 
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Question  13:  What  is  your  primary  state  of  residence? 


50 

45 

40 

35 

30 

25 

20 

15 

10 

5 

0 


i  i i  i  r 

TO  to  to  O 

£  o'c"S 

■gc£o 

<T3 - 


i  i  i  i  r 


T 


,  I ,  M  ,  ■  ,  Ml 


TO'Z>totov/1"Ov/1Cto'o'C  TO  TO  >*  o  TO  O  c  ^  ^  TO 

'§'1  s  §  |  §S  as  sgig'g  as si 


-2<=o 

<  QS 


ti  O  To 

3  S3 
Bio 

c  ri 
c  Q 

O 

u 


CA 


- - ,,  . - :  tn  o 

"-Sflsi 

8  55 

V) 

TO 


Qj  >  2!  _£- 
20  £ 

^  O 


o  > 


c 

c 

o 

Q_ 


c  o 
o  ^ 

w  c 
CUD  O 
c  Q_ 
"P  wo 
O 


IDE 
IVA 
I  MO 
IMD 
ICA 


wo 


MD 


MO 


Alabama 

0 

■■ 

1 

■■ 

0 

1 

■■ 

0 

2 

1.27% 

Arizona 

1 

0 

0 

0 

0 

1 

0.64% 

California 

36 

0 

■■ 

8 

■■ 

0 

0 

44 

28.03% 

Colorado 

0 

0 

3 

0 

0 

3 

1.91% 

Connecticut 

0 

■■ 

0 

■■ 

0 

1 

■■ 

0 

1 

0.64% 

Delaware 

0 

0 

0 

0 

31 

31 

19.75% 

Florida 

0 

0 

■■ 

1 

1 

■■ 

0 

2 

1.27% 

Georgia 

0 

0 

0 

1 

0 

1 

0.64% 

Illinois 

0 

0 

■■ 

0 

2 

■■ 

0 

2 

1.27% 

Indiana 

0 

0 

0 

1 

0 

1 

0.64% 

Iowa 

0 

0 

■■ 

2 

■■ 

0 

■■ 

0 

2 

1.27% 

Kansas 

0 

0 

1 

0 

0 

1 

0.64% 

Maryland 

0 

21 

■■ 

0 

3 

■■ 

1 

25 

15.92% 

Massachusetts 

0 

0 

0 

1 

0 

1 

0.64% 

Michigan 

0 

0 

1 

0 

0 

1 

0.64% 

Minnesota 

0 

0 

5 

0 

0 

5 

3.18% 

Mississippi 

0 

■■ 

0 

■■ 

1 

0 

■■ 

0 

1 

0.64% 

Missouri 

0 

0 

2 

0 

0 

2 

1.27% 

Nebraska 

0 

0 

■■ 

1 

0 

■■ 

0 

1 

0.64% 

Nevada 

1 

0 

0 

0 

0 

1 

0.64% 

New  Jersey 

0 

0 

■■ 

0 

0 

2 

2 

1.27% 

New  Mexico 

0 

0 

2 

0 

0 

2 

1.27% 

New  York 

0 

■■ 

0 

1 

3 

■■ 

0 

4 

2.55% 

North  Carolina 

0 

0 

0 

1 

0 

1 

0.64% 

Ohio 

0 

0 

■■ 

0 

1 

■■ 

0 

1 

0.64% 

Oregon 

0 

0 

1 

0 

0 

1 

0.64% 

Pennsylvania 

0 

0 

0 

4 

■■ 

0 

4 

2.55% 

Texas 

0 

0 

1 

1 

0 

2 

1.27% 

Virginia 

0 

0 

■■ 

0 

■■ 

2 

■■ 

0 

2 

1.27% 

Washington 

0 

0 

5 

0 

0 

5 

3.18% 

No  Response 

0 

1 

■■ 

2 

■■ 

1 

■■ 

1 

5 

3.18% 

VA 


DE 


Total 


Percent 


*  States  not  listed  had  zero  responses. 
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Question  14:  Which  best  describes  your  ethnicity? 


■  ■ 

CA 

MD 

MO 

VA 

DE 

Total 

Percent 

American  Indian  or  Alaska 
Native 

0 

1 

0 

0 

0 

1 

0.64% 

Asian 

14 

3 

8 

2 

2 

29 

18.47% 

Black  or  African  American 

i 

2 

0 

0 

4 

i 

4.46% 

Hispanic  or  Latino 

6 

0 

2 

2 

0 

10 

6.37% 

Native  Hawaiian  or  Other 
Pacific  Islander 

0 

0 

0 

0 

0 

0 

0.00% 

White 

16 

15 

24 

18 

27 

100 

63.69% 

No  Response 

i 

2 

3 

2 

2 

10 

6.37% 
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Question  15:  Which  option  best  describes  your  current  level  of  education? 


Currently  in  pre-High  School 
Currently  in  High  School 
Completed  High  School 
Currently  in  Undergrad  Program 
Completed  Undergrad  Program 
Currently  in  Graduate  Program 
Completed  Graduate  Program 
Currently  in  Doctoral  Program 
Completed  Doctoral  Program 
No  Response 


20 


40 


60 


80 


ICA 
IMD 
I  MO 
IVA 
IDE 


J 


CA 

MD 

MO 

■  VA 

DE  ■ 

Total 

Percent 

Currently  in  pre-High  School 

0 

0 

0 

0 

0 

0 

0.00% 

Currently  in  High  School 

0 

16 

0 

0 

0 

16 

10.19% 

Completed  High  School 

5 

3 

2 

1 

1 

12 

7.64% 

Currently  in  Undergrad 
Program 

21 

0 

14 

12 

20 

67 

42.68% 

Completed  Undergrad 
Program 

8 

1 

5 

6 

5 

25 

15.92% 

Currently  in  Graduate 
Program 

2 

0 

8 

1 

6 

17 

10.83% 

Completed  Graduate 

Program 

2 

0 

2 

1 

1 

6 

3.82% 

Currently  in  Doctoral 

Program 

0 

0 

4 

2 

1 

7 

4.46% 

Completed  Doctoral  Program 

0 

1 

0 

o 

0 

i 

0.64% 

No  Response 

0 

2 

2 

1 

1 

6 

3.82% 
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Question  16:  What  is  the  name  of  your  current  academic  institution?  (high 
school,  university,  college,  etc.) 

CA  ^■■SBHiSI  MO  VA  DE  Total  Percent 
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Technoh 


Saint  Cloud  State  University 


San  Diego  State  University 


San  Diego  State  University 


Santa  Clara  University 


Self  Study 


Towson  University 


Unidentifiable  or  vague  term 
or  acronym 


University  of  Akron 


University  of  Arizona 


University  of  Delaware 


University  of  Maryland 
University  College 


University  of  Missouri 


University  of  New  Mexico 


University  of  Texas  at  San 
Antonio 


University  of  Washington 

Virginia  Tech 


Western  School  of 
Technology  and 
Environmental  Science 


Whatcom  Community 
College 


Wildelake  High  school 


Wilmington  University 


15 


1 

15 


0.65% 


0.65% 


9.68% 


*  Numbers  for  like  terms  have  been  consolidated.  All  terms  in  responses  with  multiple  terms  are  included. 
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Question  1 7:  What  is  your  current  grade  point  average  (on  a  4.0  scale)?  (If 


CA 

MD 

MO 

VA 

DE 

Total 

4.1 

■ 

0 

■■ 

1 

■■ 

0 

0 

0 

1 

4 

1 

1 

6 

5 

6 

19 

3.9 

2 

0 

5 

3 

4 

■■ 

14 

3.8 

1 

0 

1 

1 

2 

5 

3.7 

4 

2 

1 

3 

1 

11 

1 

1 

3 

0 

1 

6 

3.5 

3 

2 

2 

2 

5 

14 

3.4 

3 

0 

1 

1 

0 

5 

3.3 

0 

2 

1 

3 

1 

7 

3.2 

3 

0 

2 

0 

5 

10 

3.1 

2 

■■ 

1 

1 

0 

3 

7 

3 

4 

4 

2 

3 

4 

17 

2.9 

2 

1 

0 

0 

1 

4 

2.8 

2 

1 

1 

0 

0 

4 

2.7 

1 

■■ 

0 

0 

0 

0 

1 

2.6 

2 

0 

0 

0 

0 

2 

2.5 

2 

0 

1 

0 

0 

3 

2.4 

0 

0 

2 

0 

0 

2 

2.3 

0 

■■ 

1 

0 

0 

0 

1 

2.2 

0 

0 

0 

0 

0 

0 

2.1 

0 

0 

0 

0 

0 

0 

2 

2 

0 

1 

0 

0 

3 

No  Response 

3 

■■ 

6 

■■ 

7 

3 

2 

21 

Average 

3.16 

3.28 

3.43 

3.61 

3.50 

3.40 

Min 

2.0 

2.3 

2.0 

3.0 

2.9 

2.0 

Mode 

3 

3 

4 

4 

3.5 

3 

Max 

4.0 

4.1 

4.0 

4.0 

4.0 

4.1 

*Numbers  are  rounded  to  the  nearest  significant  digit. 
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Question  18:  What  is  your  current  major  or  primary  area  of  study?  (If 
applicable) 


CA 

MD 

MO 

VA 

■  DE 

Total 

Aerospace 

0 

1 

o 

o 

1 

2 

Biochemistry 

0 

0 

0 

0 

1 

1 

Business  Administration 

0 

0 

0 

1 

o 

1 

Computer  Engineering  or 

CPE 

1 

1 

3 

0 

1 

6 

Computer  Information 

Systems  or  CIS 

13 

0 

2 

3 

5 

23 

Computer  Science 

7 

2 

8 

4 

4 

25 

Computer  Security, 
Information  Security  or 
Information  Assurance 

5 

1 

8 

11 

2 

27 

Computers  (General  Term), 
Information  Systems  or 
Information  Technology 

0 

6 

5 

1 

1 

13 

Criminal  Justice 

0 

0 

0 

i 

0 

i 

Economics 

0 

0 

1 

0 

0 

1 

Electrical  Engineering 

1 

0 

1 

0 

2 

4 

Forensics 

0 

0 

0 

3 

0 

3 

Graphic  Design 

0 

0 

0 

0 

1 

i 

Information  System  Auditing 

1 

0 

0 

0 

0 

1 

Management  Information 
Systems  or  MIS 

0 

0 

1 

0 

0 

1 

Mathematics 

0 

0 

0 

0 

1 

1 

Music  Theory 

0 

1 

0 

0 

0 

i 

Networking  or  Network 
Security 

1 

3 

3 

3 

18 

28 

None  or  N/A 

0 

i 

i 

i 

0 

3 

Programming 

1 

1 

0 

0 

0 

2 

*Numbers  for  like  terms  have  been  consolidated.  All  terms  in  responses  with  multiple  terms  are  included. 
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Question  19:  What  do  you  consider  to  be  your  two  greatest  academic  strength 


areas?  (e.g.,  math,  science,  computers,  English,  art,  etc.) 

CA 

■■ 

MD 

MO 

VA  || 

DE 

Total 

Art 

2 

0 

0 

1 

0 

3 

Biology 

1 

0 

0 

0 

0 

1 

Business  or  Management 

1 

0 

6 

0 

0 

7 

Computers 

29 

11 

29 

20 

27 

116 

Criminal  Justice 

0 

0 

0 

0 

1 

1 

Engineering 

1 

1 

0 

0 

0 

2 

English 

5 

1 

4 

6 

7 

23 

Exercise  or  Sports 

1 

0 

0 

0 

0 

1 

Foreign  Language 

0 

■■ 

1 

1 

0 

1 

3 

History 

0 

1 

0 

1 

1 

3 

Math 

8 

15 

7 

7 

13 

50 

Music 

1 

0 

1 

0 

0 

2 

Networking 

1 

0 

1 

0 

0 

2 

Philosophy 

1 

0 

0 

0 

0 

1 

Physics 

1 

1 

0 

0 

0 

2 

Programming 

1 

0 

1 

0 

0 

2 

Psychology 

1 

0 

0 

0 

0 

1 

Science 

8 

8 

8 

8 

10 

42 

Security 

1 

■■ 

0 

2 

0 

0 

3 

*Numbers  for  like  terms  have  been  consolidated.  All  terms  in  responses  with  multiple  terms  are  included. 


Question  21:  Have  you  ever  had  formal  training/  coursework  on  cyber  security 
topics? 


CA 

MD 

MO 

VA 

DE 

Total 

Yes  Responses 

21 

4 

21 

15 

18 

■I 

79 

Yes  Percentage 

55.26% 

17.39% 

56.76% 

62.50% 

51.43% 

50.32% 

No  Responses 

17 

■I 

18 

14 

8 

■■ 

15 

72 

No  Percentage 

44.74% 

78.26% 

37.84% 

33.33% 

42.86% 

45.86% 

No  (Blank)  Response 

0 

1 

2 

1 

2 

6 
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Question  22:  On  a  scale  of  1-10,  what  is  your  level  of  familiarity/  comfort  with 


the  field  of  cyber  security? 
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Question  23:  If  you  have  a  background  in  cyber  security,  what  areas  of  cyber 
security  do  you  consider  yourself  most  proficient  in?  (e.g.,  system  administration, 
programming,  forensics,  system  penetration,  etc.) 


CA 

MD 

MO 

VA 

DE 

Total 

Binary  Analysis 

1 

0 

0 

0 

0 

1 

Cryptography 

0 

0 

1 

0 

0 

1 

Forensics 

7 

1 

2 

3 

2 

15 

Information  Assurance 

0 

0 

1 

1 

0 

2 

Intrusion  Detection 

1 

0 

o 

2 

0 

3 

Networking 

1 

1 

3 

4 

1 

10 

Operating  Systems 

0 

1 

0 

0 

0 

1 

Operations  Management 

0 

0 

3 

0 

0 

3 

Penetration  Testing 

3 

1 

3 

4 

3 

14 

Programming 

3 

2 

6 

3 

5 

19 

System  Administration 

10 

5 

4 

8 

7 

34 

System  Hardening 

2 

1 

2 

0 

0 

5 

System  Hardware 

0 

1 

0 

0 

0 

1 

Traffic  Analysis 

1 

0 

0 

1 

0 

2 

User  Training 

0 

0 

i 

0 

0 

1 

Vulnerability  Assessment 

1 

0 

1 

1 

0 

3 

Web  Application  Security 

3 

0 

0 

0 

1 

4 

*Numbers  for  like  terms  have  been  consolidated.  All  terms  in  responses  with  multiple  terms  are  included. 
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Question  24:  Where  did  you  first  hear  about  the  USCC  organization? 

r  \ 

■  Email/ Newsletter 

■  Facebook 

■  Family  or  Friend 

■  Magazine  Article 

■  Newspaper  Story 

■  Professor 

■  TV  News 

■  Twitter 

■  University  Publication 

■  Website/ Search  Engine 

■  YouTube 


j 


CA 

■■ 

MD 

MO 

VA 

DE 

Total 

Percent 

1 

Email/  Newsletter 

3 

3 

12 

5 

4 

27 

17.20% 

Facebook 

0 

1 

0 

0 

1 

2 

1.27% 

Family  or  Friend 

6 

4 

2 

4 

3 

19 

12.10% 

Magazine  Article 

1 

0 

0 

0 

0 

1 

0.64% 

Newspaper  Story 

0 

■■ 

0 

0 

1 

0 

1 

0.64% 

Professor 

21 

9 

14 

2 

17 

63 

40.13% 

TV  News 

0 

0 

0 

0 

0 

0 

0.00% 

Twitter 

0 

0 

0 

2 

0 

2 

1.27% 

University  Publication 

1 

2 

2 

2 

6 

13 

8.28% 

Website/  Search  Engine 

4 

1 

2 

6 

1 

14 

8.92% 

YouTube 

1 

■■ 

0 

0 

0 

0 

1 

0.64% 

No  Response  or  Other 

1 

3 

5 

2 

3 

14 

8.92% 
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Question  25:  Have  you  engaged  in  cyber/  security  activities ,  events,  or 
competitions  in  the  past? 


CA 

MD 

MO 

VA 

DE 

Total 

Yes  Responses 

18 

■■ 

7 

15 

11 

10 

61 

Yes  Percentage 

47.37% 

30.43% 

40.54% 

45.83% 

28.57% 

38.85% 

No  Responses 

20 

■I 

12 

20 

12 

22 

■■ 

86 

No  Percentage 

52.63% 

52.17% 

54.05% 

50.00% 

62.86% 

54.78% 

No  (Blank)  Response 

0 

■I 

4 

2 

1 

■■ 

3 

10 
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Question  26:  If  you  have  engaged  in  similar  events  in  the  past ,  briefly  list  these 
previous  events  and  activities. 


CA 

MD 

MO 

VA 

DE 

Total 

Access  Data  Forensic  Boot 
Camp 

0 

0 

0 

0 

1 

1 

Cal  Poly  Cyber  Security  Fair 

1 

0 

0 

0 

0 

1 

Cal  Poly  I.T.  Competition 

2 

0 

0 

0 

0 

2 

CCDC 

3 

1 

4 

2 

1 

11 

CSC  Cyber  Battleground 

0 

0 

0 

1 

0 

1 

Cyber  Watch  Challenge 

0 

1 

0 

0 

0 

1 

CyberFoundations 

1 

0 

0 

0 

0 

1 

CyberPatriot 

2 

0 

0 

0 

0 

2 

CyberQuest 

0 

0 

1 

0 

0 

1 

DC3  Challenge 

1 

0 

0 

2 

1 

4 

Defcon 

2 

0 

2 

1 

0 

5 

Hackerspace 

0 

0 

1 

0 

0 

1 

Hackid 

0 

0 

1 

0 

0 

1 

HIMMS 

0 

0 

1 

0 

0 

1 

How  Strong  Is  Your  Fu  2  - 
Online  CTF 

0 

0 

0 

1 

0 

1 

InfraGard 

0 

0 

1 

0 

0 

1 

Iowa  State  University  Cyber 
Defense  Competition 

0 

0 

1 

0 

0 

1 

ISACA 

0 

0 

1 

0 

0 

1 

ISSA 

0 

0 

1 

0 

0 

i 

ISTS  Hack  and  Defend 

0 

0 

0 

2 

0 

2 

MACCDC 

0 

0 

0 

o 

2 

2 

Maryland  Cyber  Challenge  & 
Conference 

0 

1 

0 

0 

0 

1 

Miami  Electronic  Crimes 

Task  Force 

0 

0 

1 

0 

0 

1 

MITRE  STEM  CTF 

0 

0 

2 

0 

0 

2 

Moraine  Valley  High  School 
Cyber  Competition 

0 

2 

0 

0 

0 

2 

NCCDC 

2 

0 

1 

0 

0 

3 

Notacon 

0 

0 

1 

0 

0 

1 

NYU  Poly  CSAW  2010 

0 

0 

0 

0 

1 

1 

OWASP 

0 

0 

1 

0 

0 

1 

PRCCDC 

0 

0 

0 

0 

0 

0 

SANS 

0 

0 

1 

0 

0 

1 

SANS  NetWars 

1 

0 

0 

5 

0 

6 

Sewcure360 

0 

0 

1 

0 

0 

1 

Shmoocon 

0 

0 

0 

1 

0 

1 

Shmoocon  Hack  or  Halo 

0 

0 

0 

0 

1 

1 

SkillsUSA 

0 

1 

0 

0 

0 

1 

TracerFIRE 

0 

0 

2 

0 

0 

2 

USCC  Cyber  Camp  2010 

3 

0 

0 

0 

2 

5 

WRCCDC 

4 

0 

0 

0 

0 

4 

*Numbers  for  like  terms  have  been  consolidated.  All  terms  in  responses  with  multiple  terms  are  included. 
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Question  27:  What  is  your  desired/ planned  career  field? 


■■  CA 

MD 

MO 

VA 

■  DE 

Total 

Aerospace  Engineering 

0 

o 

0 

0 

1 

1 

Business  or  Management 

1 

0 

2 

0 

0 

3 

Computer  Engineering 

1 

3 

0 

0 

o 

4 

Computer  Science 

0 

2 

1 

0 

0 

3 

Electrical  Engineering 

0 

o 

0 

0 

2 

2 

Embedded  System  Desing 

0 

0 

0 

0 

1 

1 

Forensics 

3 

i 

0 

1 

o 

5 

Game  Development 

0 

1 

0 

0 

0 

1 

Government  or  Intelligence 

o 

0 

3 

1 

0 

4 

Incident  Response  or  Cyber 
Crime  Prevention 

1 

0 

1 

3 

0 

5 

Information  Assurance  or 
Computer  Security 

8 

6 

15 

10 

7 

46 

Information  Technology 

4 

1 

9 

1 

4 

19 

Nanotechnology 

0 

i 

0 

0 

0 

i 

Networking  or  Network 
Security 

7 

1 

2 

2 

4 

16 

Penetration  Testing 

6 

0 

2 

2 

4 

14 

Programming  or  Software 
Engineering 

2 

1 

1 

2 

2 

8 

Robotics 

i 

0 

o 

o 

0 

i 

*Numbers  for  like  terms  have  been  consolidated.  All  terms  in  responses  with  multiple  terms  are  included. 


Question  28:  Have  you  previously  considered  a  career  in  cyber  security? 


CA 

MD 

MO 

VA 

DE 

Total 

Yes  Responses 

29 

■I 

14 

29 

22 

24 

■■ 

118 

Yes  Percentage 

76.32% 

60.87% 

78.38% 

91.67% 

68.57% 

75.16% 

No  Responses 

9 

6 

6 

0 

9 

30 

No  Percentage 

23.68% 

26.09% 

16.22% 

0.00% 

25.71% 

19.11% 

No  (Blank)  Response 

0 

■I 

3 

2 

2 

2 

9 
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Question  29:  On  a  scale  of  1-5,  after  your  participation  in  this  event  is  what  is 
the  likelihood  you  will  to  explore  cyber  security  as  a  career  path? 


CA 


MD  ■■  MO 


VA 


DE 


Total 


Percent 


1,  Not  At  All 


1.27% 


2,  Unlikely 


3.82% 


3,  Likely 


10 


27 


17.20% 


4,  Very  Likely 


12 


14 


44 


28.03% 


5,  Certainly 


21 


16 


17 


11 


69 


43.95% 


No  Response 


5.73% 
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Summary  2010  Pilot  Participant  Survey  Results  Analysis 


Summary:  Total  Survey  Completions 


NY 

DE 

CA 

TOT 

Number* 

15 

19 

21 

55 

Percent 

27.27% 

34.55% 

38.18% 

100.00% 

*Number  based  on  largest  single  day  survey  count. 
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2010  New  York  Camp  Results  Summary 


Q1 

■  Day  1 

■  Day  2 

■  Day  3 

■  Day  4 


123456789  10 


Q2 

■  Day  1 

■  Day  2 

■  Day  3 

■  Day  4 


123456789  10 


Q3 


■  Day  1 

■  Day  2 

■  Day  3 

■  Day  4 


123456789  10 


2010  New  York  Camp 

Q1  -  Overall 

02  -  Teaching  Skill 

03  -  Value  of  Course 

Day  1  AVG 

9.33 

9.93 

9.20 

Day  1  MIN 

7.00 

9.00 

5.00 

Day  1  MAX 

10.00 

10.00 

10.00 

Day  1  MODE 

10.00 

10.00 

10.00 

Day  2  AVG 

9.82 

9.91 

9.91 

Day  2  MIN 

9.00 

9.00 

9.00 

Day  2  MAX 

10.00 

10.00 

10.00 

Day  2  MODE 

10.00 

10.00 

10.00 

Day  3  AVG 

9.55 

9.55 

9.45 

Day  3  MIN 

7.00 

7.00 

7.00 

Day  3  MAX 

10.00 

10.00 

10.00 

Day  3  MODE 

10.00 

10.00 

10.00 

Day  4  AVG 

9.55 

9.64 

9.18 

Day  4  MIN 

7.00 

7.00 

7.00 

Day  4  MAX 

10.00 

10.00 

10.00 

Day  4  MODE 

10.00 

10.00 

10.00 

*Graphs  display  score  (1  -poor  to  10-  excellent)  on  the  x-axis  and  number  of  responses  at  that  score  on  they-axis. 
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2010  Delaware  Camp  Results  Summary 


16 

14 

12 

10 

8 

6 

4 

2 

0 


III 

■ 

_ 1 

1  1 

1  3 

Q1 


1  Day  1 
I  Day  2 
i  Day  3 
l  Day  4 


123456789  10 


15 

16 
14 
12 
10 

8 

6 

4 

2 

0 


“i  i  t  r 


123456789  10 


Q2 


1  Day  1 
I  Day  2 
i  Day  3 
l  Day  4 


18 
16 
14 
12 
10 
8 
6 
4 
2 
0 

123456789  10 


Q3 


■  Day  1 

■  Day  2 

■  Day  3 

■  Day  4 


2010  Delaware  Camp 

Q1  -  Overall 

02  -  Teaching  Skill 

03  -  Value  of  Course 

Day  1  AVG 

8.68 

9.42 

8.84 

Day  1  MIN 

8.00 

9.00 

7.00 

Day  1  MAX 

10.00 

10.00 

10.00 

Day  1  MODE 

9.00 

9.00 

10.00 

Day  2  AVG 

9.06 

9.25 

9.19 

Day  2  MIN 

5.00 

7.00 

4.00 

Day  2  MAX 

10.00 

10.00 

10.00 

Day  2  MODE 

10.00 

9.00 

10.00 

Day  3  AVG 

9.68 

9.84 

9.89 

Day  3  MIN 

8.00 

8.00 

9.00 

Day  3  MAX 

10.00 

10.00 

10.00 

Day  3  MODE 

10.00 

10.00 

10.00 

Day  4  AVG 

8.88 

9.25 

9.06 

Day  4  MIN 

7.00 

6.00 

7.00 

Day  4  MAX 

10.00 

10.00 

10.00 

Day  4  MODE 

9.00 

10.00 

9.00 

*Graphs  display  score  (1  — poor  to  10  —  excellent)  on  the  x-axis  and  number  of  responses  at  that  score  on  they-axis. 
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2010  California  Camp  Results  Summary 


123456789  10 


Q2 


■  Day  1 

■  Day  2 

■  Day  3 

■  Day  4 


Q3 

■  Day  1 

■  Day  2 

■  Day  3 

■  Day  4 


123456789  10 


2010  California  Camp 

Q1  -  Overall 

02  -  Teaching  Skill 

03  -  Value  of  Course 

Day  1  AVG 

9.10 

9.24 

■■ 

9.24 

Day  1  MIN 

7.00 

6.00 

8.00 

Day  1  MAX 

10.00 

10.00 

■■ 

10.00 

Day  1  MODE 

10.00 

10.00 

10.00 

Day  2  AVG 

8.55 

9.75 

8.45 

Day  2  MIN 

2.00 

8.00 

2.00 

Day  2  MAX 

10.00 

10.00 

■■ 

10.00 

Day  2  MODE 

8.00 

10.00 

10.00 

Day  3  AVG 

9.05 

8.75 

9.10 

Day  3  MIN 

4.00 

5.00 

4.00 

Day  3  MAX 

10.00 

10.00 

10.00 

Day  3  MODE 

10.00 

10.00 

10.00 

Day  4  AVG 

9.35 

9.45 

9.55 

Day  4  MIN 

8.00 

6.00 

8.00 

Day  4  MAX 

10.00 

10.00 

10.00 

Day  4  MODE 

10.00 

10.00 

10.00 

*Graphs  display  score  (1  — poor  to  10  —  excellent)  on  the  x-axis  and  number  of  responses  at  that  score  on  they-axis. 
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Background 

In  previous  USCC  summer  camps,  Capture  the  Flag  (CTF)  environments  were  built  with  support 
of  various  sponsors  which  while  successful  were  deemed  not  to  be  a  scalable  CTF  environment. 
As  a  result,  an  “Issue  Paper  for  CTF”  was  written  to  explore  the  question  ‘which  virtual 
environment  should  be  used  for  the  “Capture  the  Flag  (CTF)”  for  the  camps?  And  what  should 
be  the  partnership(s)  for  the  future?”  The  paper  recommended  evaluating  the  following  three 
environments  in  the  USCC  2011  Summer  Camps. 

•  SAIC’s  CyberNEXS™:  SAIC’s  CyberNEXS™  is  a  third  generation  cyber  training  and 
exercise  environment.  Using  virtualization,  CyberNEXS  games  emulate  an  information 
enterprise  consisting  of  live  Windows  and  UNIX  operating  systems,  and  network  and 
security  devices.  Self-contained,  it  quantifies  performance  in  key  cyber  skills,  including 
hardening  of  systems,  maintenance  of  critical  services,  recognizing  and  thwarting  attacks, 
forensics,  penetration  testing  and  communications.  CyberNEXS  is  highly  scalable  to 
hundreds  of  contestants  and  can  be  tailored  to  meet  the  training  and  exercise 
requirements  for  High  School,  Collegiate  and  Professional  levels  of  game  play. 
Conducted  via  the  Internet,  CyberNEXS  has  been  used  around  the  world,  providing  real¬ 
time,  high  fidelity  scoring  in  support  of  Defense,  Federal  and  Commercial  training  and 
exercise  events. 

•  iSIGHT  Partners:  The  iSIGHT  Partners  solution  is  the  leading  technology  platform  for 
high  end  cyber  exercises.  This  proven  technology  and  team  have  been  delivering 
competitions  around  the  world  since  2005.  The  iSIGHT  Partners  solution  open  exercise 
framework  creates  a  high  degree  of  realism.  This  realism  combined  with  unique  scoring 
capability  provides  an  excellent  evaluation  of  real  world  cyber  security  skills  and 
technology.  The  open  framework  supports  a  wide  array  of  target  technologies  and  attack 
vectors  including  IPv4  and  IPv6,  wireless  (802.1 1,  RFID,  GSM),  VoIP  and  even 
SCADA.  Over  the  years  the  team  and  technology  has  delivered  exercises  for  an  ever 
growing  array  of  private  and  public  customers  around  the  world. 

•  CSSIA:  The  Center  for  Systems  Security  and  Information  Assurance  (CSSIA)  has  a  CTF 
technical  environment  hosted  at  Moraine  Valley  Community  College  outside  of  Chicago, 
IL.  This  effort  is  funded  by  the  National  Science  Foundation. 

As  a  result,  the  201 1  USCC  Summer  Camps  utilized  the  following  environments  for  their  CTF  exercises: 


Camp 

2011  Tool  Used 

California 

SAIC  CyberNEXS 

Maryland 

CSSIA 

Missouri 

SAIC  CyberNEXS 

Virginia 

iSIGHT  Partners 

Delaware 

iSIGHT  Partners 
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Survey  Structures 

2011  CTF  Survey  Structure 

5.  Event  Setup 

5.1.  Was  the  vendor  helpful  in  explaining  the  set  up  for  this  event?  [yes/  no  selection  w/  detail  memo 
field  option] 

5.2.  On  a  scale  of  1-5,  how  would  you  rate  the  quality  of  the  technical  documentation  provided  by  the 
vendor?  [drop  down  menu  w/  detail  memo  field  option] 

5.2.1. 1,  Inadequate 

5.2.2. 2,  Poor 

5. 2. 3. 3,  Acceptable 

5.2.4. 4,  Good 

5.2. 5. 5,  Excellent 

5.3.  Did  the  competition  appear  to  provide  the  technical  challenge  for  the  class?  [yes/  no  selection  w/ 
detail  memo  field  option] 

5.4.  Do  you  feel  the  technical  set  up  was  easy  to  achieve?  [yes/  no  selection] 

5.5.  Did  the  technical  performance  meet  your  expectations?  [yes/  no  selection] 

6.  Evaluation 

6.1.  On  a  scale  of  1-10,  what  is  your  overall  evaluation  of  this  course/  event?  [drop  down  menu] 

6.1. 1.1,  Bad 

6. 1.2. 2, 

6. 1.3. 3,  Poor 

6. 1.4.4, 

6. 1.5. 5,  Marginal 

6. 1.6.6, 

6. 1.7.7,  Good 

6. 1.8. 8, 

6. 1.9. 9,  Great 

6.1.10.  10,  Excellent 

6.2.  On  a  scale  of  1-10,  what  is  your  evaluation  of  the  scoreboard  display?  [drop  down  menu] 

6.2.1. 1,  Bad 

6.2. 2. 2, 

6.2. 3. 3,  Poor 

6.2.4. 4, 

6.2. 5. 5,  Marginal 

6.2.6. 6, 

6.2.1. 1,  Good 
6.2.8. 8, 

6.2.9. 9,  Great 

6.2.10.  10,  Excellent 

6.3.  On  a  scale  of  1-10,  what  is  your  evaluation  of  the  network  display  for  the  TAs?  [drop  down 
menu] 

6.3. 1.1,  Bad 

6. 3.2. 2, 

6. 3. 3. 3,  Poor 
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6. 3.4.4, 

6. 3. 5. 5,  Marginal 
63.6.6, 

63.1.1,  Good 
6.3.8. 8, 

63.9.9,  Great 

6.3.10.  10,  Excellent 

6.4.  On  a  scale  of  1-10,  what  is  your  evaluation  of  the  ease  of  use  for  the  technical  solution?  [drop 
down  menu] 

6.4. 1.1,  Bad 

6.4.2. 2, 

6.4.3. 3,  Poor 

6.4.4. 4, 

6.4.5. 5,  Marginal 

6.4.6. 6, 

6.4.7. 7,  Good 

6.4.8. 8, 

6.4.9. 9,  Great 

6.4.10.  10,  Excellent 

7.  Event  Assessment 

7. 1 .  Would  you  use  this  solution  for  your  upcoming  activities?  [free  form  memo  field] 

7.2.  What  are  the  strengths  of  this  competition?  [free  form  memo  field] 

7.3.  What  are  the  areas  for  improvements  for  this  event/  competition?  [free  form  memo  field] 

7.4.  Would  you  recommend  this  event/  competition  to  your  peers?  [yes/  no  selection] 

8.  Closing 

8.1.  If  you  have  any  additional  comments  or  feedback,  please  provide  it  here,  [free  form  memo  field] 
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Comprehensive  2011  CTF_  Survey  Results  Analysis 

As  part  of  the  2011  Camps,  USCC  requested  that  camp  hosts  complete  a  technical  survey  related  to  the 
Capture  the  Flag  activity  used  at  their  camp.  The  USCC  designed  the  survey  questions  to  assess  the 
ability  of  the  camp  hosts  to  set  up  the  virtual  CTF  and  how  easy  it  would  be  to  use.  In  the  case  of 
Delaware,  the  camp  did  not  provide  instructions  to  the  TAs  or  students  regarding  the  surveys,  therefore, 
many  of  the  surveys  were  completed  after  the  CTF  which  led  to  incomplete  results 

Furthermore,  in  reviewing  the  submitted  CTF  data,  a  number  of  important  observations  should  be  taken 
into  account  regarding  the  results.  The  California,  Maryland,  and  Missouri  camps  requested  the  CTF 
Survey  responses  from  camp  TA’s,  hosts  and  administrators  only  whereas  the  Virginia  camp  had  campers 
complete  the  CTF  survey  as  well  as  the  TAs,  hosts  and  administrators.  There  were  no  responses  from  the 
Delaware  camp.  In  reviewing  the  results  the  nature  of  the  respondents  as  well  as  the  specific  tool  used  at 
each  camp  should  be  taken  into  consideration. 


Summary:  Total  Survey  Responses 


CA 

MD 

MO 

VA 

DE 

Total 

Number 

2 

4 

2 

15 

0 

23 

Percent 

8.70% 

17.39% 

8.70% 

65.22% 

0.00% 

100.00% 
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Question  1:  Was  the  vendor  helpful  in  explaining  the  set  up  for  this  event? 


CA 

MD 

MO 

VA 

DE 

Total 

Yes  Responses 

2 

4 

i 

11 

0 

■■ 

18 

Yes  Percentage 

100.00% 

100.00% 

50.00% 

78.57% 

0.00% 

81.82% 

No  Responses 

0 

II 

0 

i 

3 

0 

■■ 

4 

No  Percentage 

0.00% 

0.00% 

50.00% 

21.43% 

0.00% 

18.18% 

No  (Blank)  Response 

0 

0 

0 

1 

0 

■■ 

i 

Comments 

•  (CA)  There  were  some  discussions  regarding  the  best  methodology.  Additionally,  they  were  available  for 
conference  calls  in  preparation  for  the  events. 

•  (CA)  The  vendor  did  provide  some  documentation.  However,  we  did  run  into  a  problem  with  Win7  64bit. 
The  remote-desktop  client  in  Win7  64bit  hosts  without  Service  Pack  1  was  incompatible  with  the 
competition  environment.  Installing  SP1  corrected  the  issue.  The  distributed  packet  documentation 
differed  from  what  was  given  to  the  teams  versus  what  was  available  in  the  game  portal,  which  led  to  some 
confusion  (the  packet  documentation  was  correct,  the  web  portal  looked  like  it  was  for  a  different  scenario 
entirely). 

•  (MD)  I  was  not  directly  involved  in  the  setup  for  this  event.  However,  from  everything  I  observed  the 
vendor  and  the  instructor  running  the  event  worked  together  smoothly. 

•  (MD)  The  vendor  walked  me  through  the  technical  details  of  their  setup  as  well  as  allowed  me  a  hands-on 
demonstration  in  addition  to  observing  the  competitors. 

•  (MO)  However,  when  testing  the  night  before,  we  needed  to  ’convince”  the  vendor  there  was  a  problem, 
they  said  it  was  the  network  and  we  had  to  provide  evidence  it  was  their  resources 

•  (MO)  Documentation  was  too  verbose,  and  lousy.  Staff  was  difficult  to  get  a  hold  of  and  stopped  returning 
phone  calls  until  we  started  including  Karen  on  the  e-mail  thread. 

•  (VA)  The  vendor  has  premeetings  but,  this  one  was  done  differently  with  the  vendor  being  on  site 

•  (VA)  Just  barely  though.  A  handout  would  have  helped  greatly. 

•  (VA)  I  presume  the  ’’vendor”  in  this  instance  is  the  group  brought  in  on  Friday  to  run  the  CTF  event.  The 
team  did  a  good  job  in  explaining  the  constraints  and  objectives  for  the  competition.  The  guidelines  were 
clear,  concise,  and  fairly  well  planned  out.  Also,  the  team  remained  accessible  throughout  the  competition. 

•  (VA)  They  did  a  great  job  with  presenting  the  rules  and  how  everything  was  being  recorded. 

•  (VA)  The  individual  courses  were  fine  but  did  not  provide  a  framework  for  thinking  about  security  issues 

•  (VA)  The  instructions  were  clear. 
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•  (VA)  Yes  the  vendor  was  helpful.  The  instructions  about  the  range  of  IPs  allowed  to  attack,  about  team 
leaders,  scoring,  etc.,  were  clear.  The  set  up  needed  was  minimal,  almost  non-existent,  we  just  connected 
the  cables  and  everything  was  up  and  running. 

•  (VA)  Tim  was  very  succinct  and  didn’t  mince  his  words,  as  usual.  :D 

•  (VA)  The  vender  explained  the  scope  of  the  CTF  and  what  was  out  of  bounds. 

•  (VA)  Registration  and  day  one  had  most  people  standing  in  a  big  group  asking  each  other  where  we  where 
supposed  to  go,  and  what  we  we’re  supposed  to  be  doing 

•  (VA)  Everything  went  very  smoothly  as  well  as  did  a  great  job  keeping  to  our  schedule.  SANS  did  a  great 
job  crunching  everything  into  a  small  time  frame  while  still  keeping  everything  very  easy  to  follow  and 
understand. 

•  (VA)  Yes,  the  vendor  spent  a  good  45  minutes  before  the  event  explaining  how  things  would  work  and  how 
the  scoring  worked.  In  addition  he  did  a  PowerPoint  presentation  explaining  the  same  information  to  the 
campers. 
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Question  2:  On  a  scale  of  1-5,  how  would  you  rate  the  quality  of  the  technical 
documentation  provided  by  the  vendor? 


CA 

MD 

MO 

VA 

DE 

Total 

Percent 

1,  Inadequate 

0 

0 

0 

2 

0 

2 

8.70% 

2,  Poor 

0 

0 

1 

0 

0 

1 

4.35% 

3,  Acceptable 

2 

2 

1 

1 

0 

6 

26.09% 

4,  Good 

0 

0 

0 

7 

0 

7 

30.43% 

5,  Excellent 

o 

1 

0 

5 

0 

6 

26.09% 

No  Response 

0 

1 

0 

0 

0 

1 

4.35% 

Comments 

•  (CA)  The  document  for  the  users  is  very  long.  Many  of  the  participants  did  not  read  it  all.  One  TA  redid 
the  document  for  ease  of  use  for  the  campers. 

•  (CA)  The  key  piece  of  missing  information  was  related  to  the  Service  Pack  requirements  for  Win7  64bit. 
Otherwise  the  documentation  was  relatively  accurate  and  complete.  Some  confusion  was  found  during  the 
registration  for  the  flag  system,  and  the  ticketing  system  does  not  specify  which  set  of  credentials 
(registered  attacker  creds  vs  VPN  creds)  to  use  when  logging  in,  which  led  to  more  confusion.  Also,  SSH 
on  the  BackTrack  hosts  was  broken  at  the  beginning  of  the  competition,  and  required  manual  intervention 
from  a  working  RD  session  to  get  working  after  receiving  the  proper  commands  from  SAIC. 

•  (MD)  Since  the  vendor  sent  someone  to  the  site  to  help  run  the  event  there  were  no  notable  technical 
hiccups. 

•  (MD)There  was  not  much  physical  documentation,  but  the  usage  was  straight  forward.  There  was  not 
highly  complicated  instructions  on  how  to  interact  with  the  environment. 

•  (MO)  Note;  the  comment  regarding  the  TA  redoing  the  documentation  is  really  for  the  MO  camp  not  the 
CA  camp.  The  documentation  is  long. 

•  (MO)  Documentation  was  15  pages  of  mostly  useless  screenshots.  While  it  provided  some  details  of  what 
was  happening,  it  also  left  many  key  questions  out  from  the  campers. 

•  (VA)  If  you  take  into  account  the  presentation  on  Friday  morning.  The  paper  documents  were  not 
sufficient  on  their  own 

•  (VA)  There  was  none. 

•  (VA)  There  were  no  documented  rules  to  review  prior  to  the  event.  The  documentation  that  we  did  receive 
had  incorrect  information  in  it... 

•  (VA)  They  did  not  provide  any  technical  documentation  at  all. 
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•  (VA)  The  vendor  did  not  provide  technical  documentation,  because  technical  documentation  was  not 
needed.  The  vendor  said:  here  are  the  IP  addresses,  this  is  how  you  get  points. 

•  (VA)  There  was  a  slight  lack  of  documentation  on  purpose  which  encouraged  students  to  figure  problems 
out  on  their  own  without  having  someone  hold  their  hands  in  the  process. 

•  (VA)  No  documentation  was  provided,  just  quick  slides.  Had  to  count  on  hand  written  notes. 

•  (VA)  Very  informative  and  well  prepared.  Didn’t  have  too  many  problems  keeping  up  with  the  lessons  even 
though  they  were  pretty  fast  paced,  everything  was  very  well  laid  out  and  extremely  well  prepared  in 
advance.  Minor  bugs  during  the  wireless  lab  had  me  behind,  TAs  did  great  helping  me  solve  my  issue,  but 
they  had  to  shut  the  lab  down,  as  some  students  were  tampering  with  the  equipment,  but  I  will  go  back  and 
try  the  lab  at  a  later  date. 

•  (VA)  The  simple  4-5  page  draft  documentation  gave  a  quick  overview  of  how  things  were  going  to  work  so 
TA’s  had  background  information.  The  PowerPoint  on  the  day  of  the  CTF  went  through  and  explained 
things  even  better. 
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Question  3:  Did  the  competition  appear  to  provide  the  technical  challenge  for 
the  class? 


CA 

MD 

MO 

VA 

DE 

Total 

Yes  Responses 

2 

3 

i 

13 

0 

■■ 

19 

Yes  Percentage 

100.00% 

100.00% 

50.00% 

92.86% 

0.00% 

90.48% 

No  Responses 

0 

II 

0 

i 

i 

0 

■■ 

2 

No  Percentage 

0.00% 

0.00% 

50.00% 

7.14% 

0.00% 

9.52% 

No  (Blank)  Response 

0 

II 

0 

0 

1 

■■ 

0 

■■ 

i 

Comments 

•  (CA)  It  was  a  little  hard  to  understand  exactly  how/what  it  is  doing  but,  the  classes  enjoyed  the 
competition. 

•  (CA)  The  more  correct  answer  is  yes  and  no.  For  those  familiar  with  OS-level  exploits,  the  challenge  level 
was  acceptable.  The  scenario  presented  had  no  application  level  exploits,  which  left  many  competitors  high 
and  dry  with  little  to  do.  Note  that  application-level  exploits  were  covered  during  the  camp  for  the  week, 
and  it  would  be  reasonable  to  expect  that  these  skills  could  be  utilized  during  the  competition.  In  my 
opinion,  the  attack  surface  was  too  small  for  the  scope  of  the  competition.  Almost  every  target  required  an 
initial  pivot  through  a  single  compromised  host.  If  a  competitor  couldn’t  determine  this  fact,  or  was  unable 
to  compromise  the  initial  host,  most  of  the  game  went  unscored. 

•  (MD)  Yes.  The  students  all  appeared  engaged  throughout  the  competition.  Since  there  was  a  tie  for  one  of 
the  categories,  perhaps  slightly  more  difficult  material  could  be  used  for  some  of  the  challenges. 

•  (MD)  The  technical  challenge  was  not  given  in  one  all-in-one  format,  as  other  solutions  might  be.  This  was 
administered  by  two  people  onsite  that  were  communicating  with  a  few  people  offsite  for  scoring.  These 
positions  were  not  overtly  challenging  and  would  not  require  highly  technical  staffing.  Some 
security/networking/sysadmin  experience  is  though  -  a  local  community  college  professor  proctored  the 
competition. 

•  (MO)  We  made  some  adjustments  on  our  part  therefore  the  teams  seemed  evenly  distributed  in  talent 

•  (MO)  The  one  person  who  really  knew  metasploit  took  half  of  the  boxes  in  the  first  45  minutes.  Those  that 
did  not  know  metasploit  were  very  frustrated.  There  was  almost  no  diversity  in  the  challenge,  it  was  all  the 
same  stuff. 

•  (VA)  This  done  due  to  the  presentation  seems  to  be  understandable  in  how  it  works 

•  (VA)  Very  Challenging.  I’d  suggest  the  challenge  would  be  difficult  for  folks  who  were  not  prepared. 

•  (VA)  The  challenges  were  really  good. 
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•  (VA)  There  was  some  confusion  as  to  if  it  was  going  to  be  a  net  war  or  capture  the  flag  format  and  what 
those  would  mean. 

•  (VA)  It  was  a  challenge.. 

•  (VA)  The  CTF  was  great,  but  not  enough  information  was  disseminated  before  or  at  the  beginning  of  the 
event. 

•  (VA)  Yes,  the  competition  was  challenging  and  the  challenges  covered  a  good  range  of  attacks  and 
techniques,  from  SQL  injection  to  privilege  escalation,  reverse  engineering,  and  so  on.  Even  the  group  that 
got  first  managed  to  get  only  few  of  the  targets. 

•  (VA)  The  CTF  was  great  fun.  The  wide  variety  of  targets  gave  the  ctf  diversity  for  many  people.  People 
with  no  pentesting  experience  could  play  with  other  challenges  to  score  points. 

•  (VA)  Definitely  a  challenge  for  me,  and  from  talking  to  the  other  participants,  no  matter  the  skill  level, 
everyone  was  learning  new  and  important  things. 

•  (VA)  Yes,  there  were  various  levels  of  challenges  within  the  competition.  Unlike  MO,  there  were  very  few 
frustrated  faces  throughout  the  CTF,  showing  there  were  challenges  that  were  at  least  able  to  be  worked  on. 
At  the  same  time  there  were  many  more  complicated  puzzles  that  were  never  solved. 
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Question  4:  Do  you  feel  the  technical  set  up  was  easy  to  achieve? 


CA 

MD 

MO 

VA 

DE 

Total 

Yes  Responses 

■ 

1 

■l 

3 

0 

12 

0 

■I 

16 

Yes  Percentage 

50.00% 

100.00% 

0.00% 

80.00% 

0.00% 

72.73% 

No  Responses 

1 

■1 

0 

2 

3 

■■ 

0 

■I 

6 

No  Percentage 

50.00% 

0.00% 

100.00% 

20.00% 

0.00% 

27.27% 

No  (Blank)  Response 

0 

0 

0 

0 

0 

■I 

0 

Question  5:  Did  the  technical  performance  meet  your  expectations? 


14 


■  Yes 

■  No 


CA 

MD 

MO 

VA 

DE 

Total 

Yes  Responses 

■ 

1 

■1 

3 

i 

12 

0 

■I 

17 

Yes  Percentage 

50.00% 

100.00% 

50.00% 

80.00% 

0.00% 

77.27% 

No  Responses 

1 

0 

i 

3 

0 

5 

No  Percentage 

50.00% 

0.00% 

50.00% 

20.00% 

0.00% 

22.73% 

No  (Blank)  Response 

■  1 

0 

■1 

1 

0 

0 

0 

i 
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Question  6:  On  a  scale  of  1-10,  what  is  your  overall  evaluation  of  this  course/ 
event? 


4.5 


•CA 
•MD 
*  MO 
-VA 
-DE 


M 

CA 

■■ 

MD 

■■ 

MO 

■■ 

VA 

■■ 

DE 

Total 

Percent 

1 

1,  Bad 

0 

0 

■■ 

1 

0 

0 

1 

4.35% 

2 

0 

0 

0 

1 

0 

1 

4.35% 

3,  Poor 

0 

0 

■■ 

0 

0 

0 

0 

0.00% 

4 

0 

0 

0 

1 

0 

1 

4.35% 

5,  Marginal 

1 

0 

■■ 

0 

0 

■■ 

0 

1 

4.35% 

6 

1 

0 

1 

0 

0 

2 

8.70% 

7,  Good 

0 

1 

ii 

0 

3 

■■ 

0 

4 

17.39% 

8 

0 

2 

0 

2 

0 

4 

17.39% 

9,  Great 

0 

0 

■■ 

0 

4 

■■ 

0 

4 

17.39% 

10,  Excellent 

0 

0 

0 

3 

0 

3 

13.04% 

No  Response 

0 

■■ 

1 

■■ 

0 

■■ 

1 

0 

2 

8.70% 
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Question  7:  On  a  scale  of  1-10,  what  is  your  evaluation  of  the  scoreboard 
display? 


CA 


MD 


MO 


VA 


DE 


Total 


Percent 


1,  Bad 


4.35% 


4.35% 


3,  Poor 


8.70% 


0.00% 


5,  Marginal 


8.70% 


4.35% 


7,  Good 


34.78% 


8 


0.00% 


9,  Great 


4.35% 


10,  Excellent 


21.74% 


No  Response 


8.70% 
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Question  8:  On  a  scale  of  1-10,  what  is  your  evaluation  of  the  network  display 
for  the  TAs? 


CA 


MD 


MO 


VA 


DE 


Total 


Percent 


1,  Bad 


4.35% 


4.35% 


3,  Poor 


4.35% 


4.35% 


5,  Marginal 


13.04% 


0.00% 


7,  Good 


17.39% 


8 


13.04% 


9,  Great 


4.35% 


10,  Excellent 


8.70% 


No  Response 


26.09% 
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Question  9:  On  a  scale  of  1-10,  what  is  your  evaluation  of  the  ease  of  use  for  the 
technical  solution? 


CA 


MD 


MO 


VA 


DE 


Total 


Percent 


1,  Bad 


4.35% 


0.00% 


3,  Poor 


13.04% 


0.00% 


5,  Marginal 


4.35% 


13.04% 


7,  Good 


21.74% 


8 


17.39% 


9,  Great 


4.35% 


10,  Excellent 


8.70% 


No  Response 


13.04% 
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Question  10:  Would  you  use  this  solution  for  your  upcoming  activities? 

•  (CA)  I  would  have  to  know  a  little  more  about  the  system  design.  Additionally,  I  believe  the  scoring  could 
be  improved  upon. 

•  (CA)  No.  I  would  use  the  solution  provided  for  last  year’s  competition  deployed  in  a  slightly  different  way. 

I  also  have  other  solutions  that  may  be  useful. 

•  (MD)  I  was  not  the  primary  person  running  this  competition  but  I  would  contact  the  individuals  involved  in 
running  this  competition  in  the  future  if  I  were  in  charge  of  running  a  similar  cyber  security  event  since  this 
event  went  well. 

•  (MD)  Yes,  but  not  for  highly  interactive  competitions.  This  is  a  good  environment  for  beginners,  as  well  as 
for  less  formal  activities. 

•  (MD)  yes 

•  (MO)  Yes  upon  further  understanding  the  of  the  scoring  mechanisms 

•  (MO)  I  definitely  would  not  use  this  solution  again.  The  Java  Applet  VPN  was  flaky,  the  scoreboard  and 
other  pages  kept  going  down,  the  scoring  method  was  awful,  and  there  was  no  diversity  of  the  challenges. 
There  wasn’t  a  single  webapp  in  the  challenge! 

•  (VA)  yes  but  we  are  testing  virutally  next. 

•  (VA)  Yes,  but  I  would  supply  participants  with  a  VMware  appliance  with  tools  needed  to  compete. 

•  (VA)  I  would  say  that  the  CTF  competition  is  an  exciting  challenge  for  the  camp  participants,  and  as  a 
participant  I  enjoyed  it  quite  a  bit.  Given  the  chance,  yes,  I  would  at  least  consider  using  such  an  option. 

•  (VA)  Yes 

•  (VA)  yes 

•  (VA)  no 

•  (VA)  Yes,  I  would  definitely  use  it. 

•  (VA)  If  I  was  to  arrange  a  CTF  competition,  I  would  absolutely  utilize  Tim’s  team. 

•  (VA)  No 

•  (VA)  Yes,  something  similar  could  definitely  turn  into  a  big  activity  that  could  catch  on  around  the  world. 

•  (VA)  Yes,  and  I  will  continue  to  learn  based  off  of  the  information  learned  this  week. 

•  (VA)  Yes,  without  a  doubt  I  would  use  this  company  again. 
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Question  11:  What  are  the  strengths  of  this  competition? 

•  (CA)  We  could  run  the  competition  virtually  and  the  resources  were  available  throughout  the  competition. 

•  (CA)  Other  than  the  client  VPN  installation,  there  is  little  setup  involved  for  the  hosting  site  prior  to  the 
competition.  The  automated  flag  system  could  use  some  work  in  terms  of  user-friendliness,  but  it  did  work. 

•  (MD)  The  opportunity  for  the  students  to  work  together  and  interact  with  live  machines. 

•  (MD)  It  was  flexible  -  the  competitors  were  not  as  skilled  as  others,  so  the  competition  was  tailored  to  meet 
their  needs  on  the  fly.  It  was  not  highly  competitive  either  -  the  competitors  appeared  to  enjoy  themselves. 
The  organization  also  tested  on  a  broad  array  of  topics  that  are  sometimes  excluded  from  other  competitive 
environments. 

•  (MD)  Actually  measuring  skills  covered  in  preceding  days  of  summer  camp 

•  (MO)  It  does  run  virtually  and  the  vendor  is  supportive  of  their  services 

•  (MO)  None  at  all.  I  was  greatly  disappointed. 

•  (VA)  There  are  good  visuals  and  the  explanations  along  with  the  flexibility  of  scoring  seemed  to 
accommodate  concerns  raised  by  camp  hosts 

•  (VA)  Folks  who  are  experienced,  skilled,  and,  practiced  will  really  stand  out. 

•  (VA)  The  way  the  teams  were  used  was  done  very  well.  It  provided  a  breadth  of  skills  and  skill  levels  to 
each  team,  and  the  anonymous  assignment  prevented  intentionally  stacked  teams  and  encouraged 
communication  and  outreach. 

•  (VA)  More  realistic  and  more  fun  when  placed  in  competitive  environment 

•  (VA)  Very  challenging,  not  too  hard  to  accomplish,  great  music. 

•  (VA)  very  challenging 

•  (VA)  the  network  and  challenges  were  good,  but  the  service  providers  did  not  give  much  information  or 
have  anywhere  you  could  re-check  would  they  said  if  you  had  any  questions  about  the  instructions. 

•  (VA)  The  variety  of  challenges  is  a  strength.  It  offers  more  opportunities  to  score.  The  team  set  up  is  also 
good,  different  team  members  can  focus  on  different  objectives. 

•  (VA)  The  wide  range  of  challenges  in  various  field  of  computer  security 

•  (VA)  Good  structure  and  organized 

•  (VA)  Range  of  networks  with  unknown  vulnerabilities  that  people  can  exercise  theory  and  have  fun  on. 

•  (VA)  Great  networking  skills  with  peers  as  well  as  people  in  the  field.  Great  learning  experience.  Great 
way  to  find  out  what  others  are  doing  and  how  they  do  it. 

•  (VA)  There  was  a  wide  diversity  in  both  types  of  challenges,  and  complexity  of  these  challenges. 
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Question  12:  What  are  the  areas  for  improvements  for  this  event/  competition? 

•  (CA)  I  think  the  scoring  and  visuals  could  be  enhanced 

•  (CA)  Better  OS  compatibility  information.  Larger  attack  surface  in  the  scenario.  Application-level 
exploits  should  be  available  in  the  competition.  Testing  scenario  should  better  represent  the  actual 
competition  setup.  The  problems  with  Win7  were  not  realized  until  the  day  of  the  competition  because  the 
test  environment  did  not  match  with  the  production  environment. 

•  (MD)  Ability  to  scale  the  difficulty  of  the  different  challenges  easily  to  account  for  different  ability  levels 
between  different  groups  of  students. 

•  (MD)  There  is  no  realtime  scoreboard  or  displays  for  the  public.  If  it  could  improve  its  automation,  that 
would  also  be  helpful.  It  also  is  not  scalable  for  a  single  event  -  one  that  might  host  1000  players. 

•  (MD)  More  content  designed  around  security 

•  (MO)  visuals.  You  have  to  keep  updating  yourself  vs  automatically  with  the  updating  of  the  scoring 

•  (MO)  1)  Use  a  real  VPN  client  like  Cisco,  rather  than  a  Java  client  that  is  flaky  even  within  a  uniform 
environment  like  a  computer  lab.  2)  Get  systems  built  to  handle  the  expected  load  of  people  (30).  It  was 
ridiculous  how  often  the  scoring  system,  scoreboard,  or  the  entire  VPN  went  down.  3)  The  vendor  should 
test  their  own  environment  beforhand.  A  simple  NMAP  scan  of  a  single  host  and  limited  ports  should  not 
have  taken  well  over  30  minutes,  and  the  VENDOR  should  have  found  this,  not  us.  Its  a  good  thing  we 
insisted  on  testing  on  Thursday  or  the  entire  event  would  have  been  a  loss.  4)  Put  diversity  into  the 
environment,  of  different  types  of  challenges  and  different  levels  of  challenges.  Everything  should  not  be 
about  metasploit  on  unpatched  systems.  There  should  be  web  apps  with  SQL  Injection,  command  injection, 
unprotected  admin  pages,  etc.  There  should  be  FTP  with  vulnerabilities,  and  various  levels  of  other  issues. 

It  shouldn’t  be  all  about  autopawn.  4)  Get  rid  of  the  need  to  know  the  host  name  to  submit  a  ticket. 

Students  were  extremely  frustrated  that  they  couldn’t  submit  anything  until  they  knew  which  host  name 
went  along  with  the  IP  they  attacked.  Add  in  the  fact  that  reverse  DNS  was  broken  in  the  environment,  and 
no  one  even  had  a  good  way  to  determine  the  host  name  without  cheating  and  looking  at  the  ’’host  status” 
display,  which  was  never  officially  told  to  them  existed.  5)  Change  the  scoring  system  so  that  getting  root 
gives  a  lot  more  points  than  everything  else.  Once  you  have  root,  you  can  get  the  rest  of  the  points.  As  a 
result  if  its  not  weighted  this  way,  getting  root  causes  you  to  win  the  competition.  6)  Fix  bugs  in  the 
scoring  system.  If  you’re  going  to  require  the  host  listed  on  the  scoring  system,  then  only  items  submitted 
for  that  host  should  work  to  give  you  points.  You  shouldn’t  be  able  to  submit  an  NMAP  scan  of  Host  A,  for 
all  hosts  A-Z,  and  get  points.  People  were  completely  playing  the  scoring  system  in  this  way. 

•  (VA)  The  documentation  will  need  to  be  addressed,  it  appears  they  seem  to  like  to  run  physically  on  site 
which  may  be  a  problem  for  the  future 

•  (VA)  It  should  also  be  tuned  to  be  a  fun  learning  experience  rather  than  pure  technical  muscle  flexing. 

•  (VA)  In  the  future,  the  competition  space  needs  to  be  planned  better.  In  the  auditorium,  with  hard  lines 
snaked  around  the  room  and  switches  balanced  on  armrests,  the  hardware  was  at  risk  and  there  were  safety 
concerns.  Furthermore,  all  of  the  teams  were  essentially  on  top  of  each  other  and  it  was  difficult  to  avoid 
shoulder  surfing.  I  don’t  think  that  the  switches  should  necessarily  have  been  accessible  to  the  players, 
given  some  of  the  offensive-type  behaviors  in  use. 

•  (VA)  Network  was  a  little  slow 

•  (VA)  Preventing  the  switches  from  being  attacked. 

•  (VA)  The  survey  that  was  used  to  determine  the  teams  had  very  little  to  do  with  a  player’s  ability  to 
compromise  systems  (except  the  SQL  injection  and  XSS  questions) 

•  (VA)  give  out  more  information  prior  to  the  event,  give  a  print  out  of  the  rules,  scoring,  and  expectations  at 
the  event. 

•  (VA)  From  a  technical  point  of  view,  I  cannot  think  of  anything  to  improve.  Minor  things  for  improvement 
are  related  to  environment  where  it  can  take  place.  Having  tables  for  example,  instead  of  just  seats,  one 
team  per  table.  Also,  I  would  have  liked  to  have  a  bit  more  time,  maybe  one  more  hour. 
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•  (VA)  The  auditorium  wasn’t  the  best  location  for  the  tournament.  And,  as  in  most  CTFs,  there  wasn’t 
enough  time  for  the  competition.  There  were  several  challenges  I  was  really  eager  to  try  but  didn’t  have 
time  to. 

•  (VA)  Better  gear,  systems  were  very  unstable 

•  (VA)  Longer  time  frame,  more  preparation  time  for  teams  (announcing  of  teams  earlier). 

•  (VA)  Coffee/Caffeine  at  all  times.  Let  people  know  further  in  advance  for  the  wireless  parts,  as  getting 
them  sent  in  time  cost  me  5  times  what  the  part  cost. 

•  (VA)  The  challenge  might  need  a  few  more  ’’intermediate”  puzzles.  The  point  progression  for  about  half 
the  teams  stagnated  about  1.5hrs  away  from  the  CTF  close. 


APPROVED  FOR  PUBLIC  RELEASE;  DISTRIBUTION  UNLIMITED. 

103 


uscc 

US  CYBER  CHALLENGE 

2011  CTF  Data  Collection  Analysis 


Question  13:  Would  you  recommend  this  event/  competition  to  your  peers? 


CA 

MD 

MO 

VA 

DE 

Total 

Yes  Responses 

2 

3 

i 

13 

0 

19 

Yes  Percentage 

100.00% 

100.00% 

50.00% 

100.00% 

0.00% 

95.00% 

No  Responses 

0 

II 

0 

i 

0 

0 

i 

No  Percentage 

0.00% 

0.00% 

50.00% 

0.00% 

0.00% 

5.00% 

No  (Blank)  Response 

0 

1 

0 

2 

0 

3 
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Question  14:  If  you  have  any  additional  comments  or  feedback ,  please  provide  it 
here. 

•  (CA)  I  do  appreciate  the  commitment  of  the  vendor  to  make  the  event  successful. 

•  (MD)  I  think  the  camp  provided  a  valuable  learning  experience  for  the  students  and  high  school  technology 
teachers  involved.  It  taught  relevant  material  in  an  engaging  fashion  that  kept  most  students  interested 
throughout  the  week.  Being  able  to  further  subdivide  between  students  of  different  experience  and  skill 
levels  could  help  improve  the  experience  for  both  the  students  and  teachers.  Also,  more  interaction  with 
younger  (20s  and  early  30s)  speakers  and  role  models  in  cyber  security  throughout  the  week  would  be 
beneficial  to  help  the  students  better  relate  to  people  involved  in  the  field. 

•  (MD)  Good  environment  for  beginners,  high  school  students,  and  informal  things  that  less  technical 
organizers  might  want  to  use. 

•  (MO)  This  capture  the  flag  was  a  real  disappointment,  and  required  a  lot  of  work  for  the  TA’s  as  a  result. 
This  vendor  clearly  has  not  done  this  very  much  before. 

•  (VA)  SANS  NetWars  was  a  better  experience  for  me. 

•  (VA)  I  would,  again,  like  to  thank  the  organizers,  teachers,  and  TAs  for  contributing  the  time,  effort,  and 

passion  to  put  on  an  exemplary  camp.  It  was  definitely  a  worthwhile  use  of  my  time,  and  I  would  do  it 
again  if  I  got  the  chance. 

•  (VA)  na 

•  (VA)  It  was  a  great  week,  I  learned  very  much  and,  most  importantly,  1  learned  where  to  look  for  the  rest  of 

things  that  I  do  not  know.  The  contact  with  both  the  SANS  instructors  and  other  participants  who  were 
more  knowledgeable  than  me  was  a  great  experience.  I  really  hope  this  event  is  repeated  and  spread  in 

every  state  and  major  city  of  the  US.  This  type  of  event  is  in  my  opinion  the  best  way  to  get  young  people 

interested  in  security.  With  respect  to  more  traditional  security  classes  in  many  universities,  it  shows 
practical  aspects,  tools  and  some  types  of  attacks,  it  is  a  lot  of  fun.  At  the  same  time,  seeing  how  certain 
types  of  attacks  work  in  practice,  seeing  them  actually  happen  in  front  of  your  eyes,  raises  awareness  about 
risks  in  cybersecurity  so  much  more  than  just  reading  about  them  on  the  news,  or  seeing  them  explained 
theoretically  from  10  miles  high  in  a  powerpoint  slide.  I  hope  this  event  will  be  repeated  in  the  future,  I  will 
try  to  go  again  and  will  recommend  it  to  everybody  in  my  university. 

•  (VA)  Another  great  CTF  by  Tim's  team!  Thanks  and  I  look  forward  to  the  next  one! 

•  (VA)  Great  fun,  excellent  experience.  Learned  allot  of  strengths  and  weaknesses. 

•  (VA)  This  CTF  was  extremely  professional  and  well  organized.  The  members  of  Insight  were  easy  to  get  a 
hold  of,  and  helpful  whenever  addressed. 
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SAIC  CyberNEXS 

iSIGHT  Partners 

CSSIA 

Ease  of  use 

•  Lacked  complete 
documentation 

•  Easy-to-use  out  of  the 
box  solution  once  setup 

•  Easy-to-use  out  of  the 
box  solution 

•  No  documentation  since 
the  tool  was  custom 

•  Need  technical 
knowledge  to  utilize 

Technical 

capabilities 

•  Available  as  a  virtual  or 
local  hardware 
capability,  only  virtual 
used  in  2011 

•  Very  specific  system 
requirements  to  access 
virtual  platform 

•  Available  as  a  virtual  or 
local  hardware 
capability,  virtual  and 
local  used  in  201 1 

•  Local  solution  with  on¬ 
site  SME  provided  best 
CTF  implementation 
experience 

•  Requires  deep 

knowledge  of  specific 
hardware,  operating 
system  and 

configurations  used  in 
environment 

End  User 
experience 

•  Survey  data  suggests  the 
tool  provided  a  good 
challenge 

•  Interviews  suggest  this 
tool  was  more  complex 
to  use  and  mange  for  the 
CTF  event 

•  Survey  data  suggests  the 
tool  provided  a  good 
challenge 

•  Interviews  suggest  this 
tool  was  easier  to  use  and 
mange  for  the  CTF  event 

•  Highly  customizable 
because  the  platform  is 
built  from  the  ground  up 

Scoreboard 

display 

•  Automated  scoring 
capability  included 

•  Scoring  algorithm  was 
unclear  and  not  very 
customizable 

•  Automated  scoring 
capability  included 

•  Highly  customizable 
scoring  configuration 

Scalability 

•  Very  scalable 

•  Requires  significant 
preparation  time  to 
understand  and  become 
proficient  in  the 
implementation  of  the 
tool 

•  Very  scalable 

•  Requires  preparation 
time  to  understand  and 
become  proficient  in  the 
implementation  of  the 
tool 

•  The  custom  solution  was 
built  on  donated 
hardware  and  suffered 
from  significant 
scalability  issues 

Costs 

•  Provided  free  for  USCC 
2011  use 

•  Undetermined  future 
costs 

•  Provided  at  a  small  fee 
for  USCC  2011  use 

•  Undetermined  future 

costs 

•  Built  on  donated 
hardware  for  USCC  2010 

•  Undetermined  future 

costs 
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Appendix:  Technical  Configurations 

Each  of  the  tools  piloted  in  the  2011  Camp  CTFs  had  differing  technical  specifications  and  requirements. 
Below  is  a  brief  summary  of  estimated  costs  from  the  original  “Issue  Paper  for  CTF”. 

1 .  The  CSSIA  environment  needs  to  have  equipment  in  place  and  the  estimated  cost  for  the 
equipment  is  $6,900. 

2.  i  Sight  Partners  is  a  virtual  environment  where  you  pick  the  type  of  the  equipment  and 
competitions  you  want  to  run.  The  more  complicated  the  greater  the  costs.  The  initial 
estimate  provided  was  $160,000  (This  is  their  commercial  offering  price). 

3.  SAIC  CyberNEXS  has  many  options  and  therefore  has  many  associated  costs. 

As  a  specific  example  of  the  technical  infrastructure  required  for  these  solutions,  the  SAIC  solution  had 
the  following  requirements. 

Hardware  Minimum  Requirements  are  as  follows: 

1 .  Windows/Macintosh/Linux  computer  that  is  supported  by  the  SSL  VPN  server 

2.  1  Ghz  or  higher  processor; 

3.  1  GB  RAM; 

4.  Keyboard  &  Mouse; 

5.  1024x768  or  higher  display;  and, 

6.  Network  connection  from  computer(s)  to  Internet.  (As  specified  in  Network  Requirements 
listed  below) 

Software  Requirements  are  as  follows: 

1 .  Web  Browser  (JavaScript  capable) 

2.  2SSH  Client  (Freeware); 

3.  VPN  Client  (Freeware); 

4.  Telnet; 

5.  RDC  (Remote  Desktop  Connection);  and, 

6.  VNC  (Virtual  Network  Computing)  (Freeware). 

Network  Requirements  are  as  follows: 

A.  Each  user  needs  a  network  connection  with  a  minimum  of  256Kbps  internet  connectivity  (uplink 
and  downlink)  and  under  150  ms  response  time  to  SAIC  VPN 

B.  Network  firewalls  and/or  Web  Proxies  should  permit  out-bound  SSL  VPN  connections  to 
cybernexs-vpn.saic.com. 

For  further  reference,  below  are  instructional  illustrations  of  the  technical  configuration  for  the  CTF 
environment  implemented  at  the  2011  Delaware  camp. 
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An  additional  component  of  the  CTF  platforms  was  the  scoring  engine  and  score  board  display.  Below 
are  illustrations  of  the  score  boards  from  the  2011  Camps. 


2011  California  CTF  Results 
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75.00 

0.00 

0.00 

100.00 

0.00 

0.00 

75.00 

250 

Team  3 

150.00 

0.00 

0.00 

0.00 

50.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

200 

Team  Name 

25.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

25 

TeamlO 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0 

DIsturtoedMime 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0 

Return  to  Status  DtipKy  Menu 
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uscc 

US  CYBER  CHALLENGE 


2011  CTF  Data  Collection  Analysis 


Attacker  Scores 


Rank 

Attacker  Name 

Team  Name 

Flags  Planted 

Overall  Score 

Date  Accomplished 

1 

Steve 

Team  Rocket 

1 

16100.0 

Fri  Jul  15  14:00:25  PDT  2011 

2 

rw 

mm 

0 

4700.0 

Fri  Jul  15  14:00:25  PDT  2011 

3 

Kos 

Team_l 

0 

3800.0 

Fri  Jul  15  14:00:25  PDT  2011 

4 

T 

Noble6 

0 

3450.0 

Fri  Jul  15  14:00:26  PDT  2011 

5 

_ Stacey _ 

Team  Rocket 

1 

2075.0 

Fri  Jul  15  14:00:25  PDT  2011 

6 

MisterU 

Team  Rocket 

1 

2000.0 

Fri  Jul  15  14:00:25  PDT  2011 

7 

EdwardW 

Team  Rocket 

1 

2000.0 

Fri  Jul  15  14:00:25  PDT  2011 

8 

_ grayjack _ 

Team  Rocket 

1 

2000.0 

Fri  Jul  15  14:00:25  PDT  2011 

9 

Cy  berN  inja  Fig  hterPi  lot 

Team_2 

0 

850.0 

Fri  Jul  15  14:00:26  PDT  2011 

10 

Noble6 

Noble6_Team 

0 

500.0 

Fri  Jul  15  14:00:26  PDT  2011 

11 

RED_TEAM_5 

RED_TEAM_5 

0 

250.0 

Fri  Jul  15  14:00:25  PDT  2011 

12 

rockin_mac 

Team_l 

0 

150.0 

Fri  Jul  15  14:00:25  PDT  2011 

13 

Travis 

Team_3 

0 

125.0 

Fri  Jul  15  14:00:25  PDT  2011 

14 

zen8 

mm 

0 

100.0 

Fri  Jul  15  14:00:25  PDT  2011 

15 

Team_3 

Team_3 

0 

75.0 

Fri  Jul  15  14:00:26  PDT  2011 

16 

Quaz_Wu 

Team_l 

0 

25.0 

Fri  Jul  15  14:00:25  PDT  2011 

17 

g  radius 

mm 

0 

0.0 

Fri  Jul  15  14:00:25  PDT  2011 

18 

epadmik 

Team_3 

0 

0.0 

Fri  Jul  15  14:00:25  PDT  2011 

19 

penddraig 

Team_l 

0 

0.0 

Fri  Jul  15  14:00:25  PDT  2011 

20 

icarus 

Team_2 

0 

0.0 

Fri  Jul  15  14:00:26  PDT  2011 

21 

HeroJoe 

mm 

0 

0.0 

Fri  Jul  15  14:00:26  PDT  2011 

22 

_ MacNinja _ 

Team_l 

0 

0.0 

Fri  Jul  15  14:00:26  PDT  2011 

23 

attack 

Team_3 

0 

0.0 

Fri  Jul  15  14:00:26  PDT  2011 

24 

yo 

Team_10 

0 

0.0 

Fri  Jul  15  14:00:26  PDT  2011 

25 

Curtis 

Noble6_Team 

0 

0.0 

Fri  Jul  15  14:00:26  PDT  2011 

26 

DisturbedMime 

DisturbedMime 

0 

0.0 

Fri  Jul  15  14:00:26  PDT  2011 

2011  Virginia  CTF  Attacker  Standings  from  main  scoreboard  of  scoring  engine 

Handle  Phone  Hornes  Flags  Score 
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LIST  OF  SYMBOLS,  ABBREVIATIONS,  AND  ACRONYMS 


USCC: 

Multi- State  IS  AC: 
CSIS: 

GAO: 

IT: 

#: 

%: 

TAs: 

MD: 

CTF: 


United  States  Cyber  Challenge 

Multi-State  Information  Sharing  and  Analysis  Center 

Center  for  Strategic  and  International  Studies 

General  Accountability  Office 

Information  Technology 

Number 

Percent 

Teaching  Assistants 
Maryland 
Capture  the  Flag 
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